Tokenizer is a key component offering a range of functionalities that significantly enhance the development experience in Spiral applications. Its primary role is in scanning specified directories, enabling developers to effortlessly manage and organize their codebase. This tool is particularly adept at identifying and utilizing classes based on specific interfaces or attributes, facilitating a variety of practical applications.
Automatic Route Registration: One of the classic applications is in identifying route attributes on controller actions. By doing so, it automates the process of registering routes in your application, leveraging the attributes defined in the controllers. This feature significantly reduces manual overhead and streamlines the routing mechanism.
Modular Structure Support: In projects where a modular architecture is essential, the Tokenizer excels. It can automatically detect and register modules (e.g., those added as composer packages) that implement a specific interface. This capability ensures a seamless integration and management of various modules within your application.
Attribute-Based Configuration: The tokenizer can scan for specific attributes in your codebase, enabling attribute-based configuration and behavior definition. This approach aligns with modern coding practices, where attributes are used to define aspects like dependency injection, configuration settings, and more.
Locators are like your code's searchlight. They help you find specific pieces of code.
If you're aiming to locate classes, turn to Spiral\Tokenizer\ClassesInterface
. Using this, you can search for classes
based on their name, the interfaces they implement, or the traits they incorporate.
Here's a quick example. Let's say you want to locate all classes that implement
the \Psr\Http\Server\MiddlewareInterface
interface:
use Spiral\Tokenizer\ClassesInterfac;
public function findClasses(ClassesInterface $classes): void
{
foreach ($classes->getClasses(\Psr\Http\Server\MiddlewareInterface::class) as $middleware) {
dump($middleware->getFileName());
}
}
The getClasses
method will then return an array of ReflectionClass
objects representing the classes found.
In case you're on the lookout for enumerations, the Spiral\Tokenizer\EnumsInterface
is what you need. It comes with
a getEnums
method to help you on your hunt:
use Spiral\Tokenizer\EnumsInterface;
public function findEnums(EnumsInterface $enums): void
{
foreach ($enums->getEnums() as $enum) {
dump($enum->getFileName());
}
}
If you wish to find specific interfaces, the Spiral\Tokenizer\InterfacesInterface
has got you covered. Use
its getInterfaces
method like so:
use Spiral\Tokenizer\InterfacesInterface;
public function findEnums(InterfacesInterface $interfaces): void
{
foreach ($interfaces->getInterfaces() as $interface) {
dump($interface->getFileName());
}
}
Warning
Tokenizer will ignore all the files that haveinlcude
orrequire
statements. This is because it's not safe to require such reflections. Please don't use them in your code.
Tokenizer, by default, conducts its search within the app directory. However, you might often want it to consider other
directories as well. Fortunately, customizing this is straightforward using the Spiral\Bootloader\TokenizerBootloader
.
Here's how you can specify additional directories:
use Spiral\Bootloader\TokenizerBootloader;
use Spiral\Boot\DirectoriesInterface;
class AppBootloader extends Bootloader
{
public function init(DirectoriesInterface $directories, TokenizerBootloader $tokenizer): void
{
$tokenizer->addDirectory($directories->get('vendor') . 'spiral/validator/src');
}
}
You might also want to exclude specific directories from Tokenizer's search. Here's how:
return [
'directories' => [
//...
],
'exclude' => [
directory('resources'),
directory('config'),
'tests',
'migrations',
],
];
Note
Remember, expanding the directories for class search can slow down the process. It's recommended to only add directories that are essential for your needs.
When dealing with vast directories, Tokenizer can slow down a bit. However, you can amp up its speed by using the scoped
class locator. This tool lets you divide and conquer by setting up specific search zones, which we call scopes
.
If you need to search for classes in a large number of directories, the tokenizer component may suffer from poor performance. In this case, you can use the scoped class locator to improve performance.
With the scoped class locator, you can define directories to be searched within named scopes. This allows you to selectively search only the directories that are relevant to your current task.
To use the scoped class locator, you will need to define your scopes:
Scopes can be defined in the app\config\tokenizer.php
config file.
Here is an example of how to define a scope named scopeName
that searches the app/Directory
directory:
return [
'scopes' => [
'scopeName' => [
'directories' => [
directory('app') . 'Directory',
],
'exclude' => [
directory('app') . 'Directory/Excluded',
]
],
]
];
Note
Theexclude
parameter is there for a reason. If there are parts of the directory you know you won't need, just tell Tokenizer to skip them. It'll make things even faster!
Once you've got your scopes ready, you can then instruct Tokenizer to only search within a chosen scope. It's like telling it which department to go to!
The Spiral\Tokenizer\ScopedClassesInterface
lets you do this with its getScopedClasses
method. Simply hand it the
name of the scope, and it'll return all the classes it finds in that zone.
To use the method, you will need to pass in the name of the scope
as an argument. The method will then return an
array of ReflectionClass
objects representing the classes found within that scope.
use Spiral\Tokenizer\ScopedClassesInterface;
final class SomeLocator
{
public function __construct(
private readonly ScopedClassesInterface $locator
) {
}
public function findDeclarations(): array
{
foreach ($this->locator->getScopedClasses('scopeName') as $class) {
// ...
}
}
}
For big codebases, regular scans using Tokenizer's locators can slow things down, especially when you're routinely searching for classes, enums, or interfaces. Class listeners provide a smarter approach, allowing you to listen in on and react to class discoveries without repeatedly scanning directories.
Think about having to search through a really big library every time you want a book. It's a lot of work. Now, think about someone telling you whenever a new book arrives in the library. That's similar to how class listeners work. Instead of searching the library every time, you get a heads-up when a new book (class) arrives. This is especially useful during application bootstrapping, where the initial scanning takes place, after which listeners are kept in the loop.
By default, listeners focus on classes, but you can easily configure them to cast their net wider to encompass enums and
interfaces too. To do this, you will need to add the following configuration to the app\config\tokenizer.php
file:
return [
'load' => [
'classes' => true, // Search for classes
'enums' => true, // Search for enums
'interfaces' => true, // Search for interfaces
],
];
Note
Remember, you don't have to enable all three. Tailor it to suit your project's needs. The more you enable, the slower the process will be.
To use this feature, you will need to include Spiral\Tokenizer\Bootloader\TokenizerListenerBootloader
bootloader in your project at the top of bootloader's list:
public function defineSystemBootloaders(): array
{
return [
\Spiral\Tokenizer\Bootloader\TokenizerListenerBootloader::class,
// ...
];
}
Read more about bootloaders in the Framework — Bootloaders section.
The next step is to create a listener class. This class should implement
the Spiral\Tokenizer\TokenizationListenerInterface
, which mandates two methods:
listen(\ReflectionClass $class)
: This method is called every time a class is discovered. You can add logic to
process or store information from this class.finalize()
: Think of this as the closing act. Once all classes are scanned and processed, this method is invoked.
It's a perfect spot for wrapping things up or finalizing operations based on the discovered classes.Here's an example of a listener:
use Spiral\Attributes\ReaderInterface;
final class RouteAttributeListener implements TokenizationListenerInterface
{
private array $attributes = [];
public function __construct(
private readonly ReaderInterface $reader,
private readonly RouterInterface $router
) {
}
public function listen(\ReflectionClass $class): void
{
foreach ($class->getMethods() as $method) {
$route = $this->reader->firstFunctionMetadata($method, Route::class);
if ($route === null) {
continue;
}
$this->attributes[] = [$method, $route];
}
}
public function finalize(): void
{
foreach ($this->attributes as [$method, $route]) {
$this->router->addRoute(...);
}
}
}
This listener, for instance, listens for classes with specific routing attributes and adds them to a router when the scan is complete.
To improve the performance of your application, you can use the Spiral\Tokenizer\Attribute\TargetAttribute
and Spiral\Tokenizer\Attribute\TargetClass
attributes to filter the classes and attributes that are processed by
listeners. This allows you o improve the performance of your code by filtering the classes and attributes that are
processed by listeners.
When you use the attributes to filter the classes that are processed by listeners, the component caches the filtered
classes in the runtime/cache/listeners
directory after the first bootstrapping of your application.
Caching the filtered classes provides several benefits to your application. It greatly reduces the amount of time required to process your codebase, since the class locator can load the filtered classes from cache rather than re-scanning your codebase every time your application starts up. This can help to improve the performance of your application and reduce the time required for application bootstrapping.
By default, caching of the filtered classes is disabled. If you want to enable caching, you can set
the TOKENIZER_CACHE_TARGETS
environment variable to true
.
TOKENIZER_CACHE_TARGETS=true
It allows you to filter classes based on their attributes. When you specify a target attribute, the class locator will only process classes that have that attribute. This can be useful if you have a listener that only needs to analyze a specific type of class, such as a controller class that has a specific routing attribute.
Here's an example of how to use it:
use Spiral\Tokenizer\Attribute\TargetAttribute;
#[TargetAttribute(Route::class, useAnnotations: true)]
final class RouteLocatorListener implements TokenizationListenerInterface
{
// ...
}
In this example, the RouteAttributeListener
will only process classes that have the Route
attribute. This means
that if the class locator finds a class without this attribute, it won't call the listen
method of this listener.
You can add multiple attributes to your listener class:
use Spiral\Tokenizer\Attribute\TargetAttribute;
use Spiral\Tokenizer\TokenizationListenerInterface;
#[TargetAttribute(Route::class)]
#[TargetAttribute(SymfonyRoute::class)]
class RouteLocatorListener implements TokenizationListenerInterface
{
public function listen(\ReflectionClass $class): void
{
// Do something with classes that have Route or SymfonyRoute attributes
}
}
You can also pass a second parameter useAnnotations: true
to the attribute to specify that the Tokenizer should look
for the target attribute in the class annotations as well. For example:
It works similarly to TargetAttribute
, but instead of filtering classes based on their attributes, it filters them
based on their type. This is useful if you have a listener that only needs to analyze a specific type of class, such
as controller, classes that implement a specific interface or extend a specific class.
Here's an example of how to use
use Spiral\Tokenizer\Attribute\TargetClass;
#[TargetClass(SymfonyCommand::class)]
final class CommandLocatorListener implements TokenizationListenerInterface
{
// ...
}
In this example, the listener will process all classes that extend the SymfonyCommand
. This means that if the class
locator finds a class that extends it, it will call the listen
method of this listener.
Note
You can add multiple attributes to your listener class.
To register your listener, you will need to use the Spiral\Tokenizer\TokenizerListenerRegistryInterface
.
Here is an example of how to register a listener:
use Spiral\Tokenizer\TokenizerListenerRegistryInterface;
class AppBootloader extends Bootloader
{
public function init(
TokenizerListenerRegistryInterface $listenerRegistry,
RouteAttributeListener $listener
): void {
$listenerRegistry->addListener($listener);
}
}
Warning
To ensure that your listeners are called correctly, make sure to register them in bootloaders from within theLOAD
section of the application Kernel. Listeners will not be called if you register them within theAPP
kernel section.
Want to know how the tokenizer is set up? Use the tokenizer:info
command.
Just run the following command:
php app.php tokenizer:info
Included directories:+------------------------------------+-------+| Directory | Scope |+------------------------------------+-------+| /vendor/intruforce/grpc-shared/src | || app/ | |+------------------------------------+-------+Excluded directories:+-----------+-------+| Directory | Scope |+-----------+-------+Loaders:+------------+------------------------------------------------------------------------------+| Loader | Status |+------------+------------------------------------------------------------------------------+| Classes | enabled || Enums | disabled. To enable, add "TOKENIZER_LOAD_ENUMS=true" to your .env file. || Interfaces | disabled. To enable, add "TOKENIZER_LOAD_INTERFACES=true" to your .env file. |+------------+------------------------------------------------------------------------------+Tokenizer cache: disabledTo enable cache, add "TOKENIZER_CACHE_TARGETS=true" to your .env file.