690991c526
Major refactoring cleanup: - Add new controller architecture (class-controller-*.php) - Add new settings-v2 UI (views/settings-v2/) - Add new CSS architecture (agentic-sidebar.css, tokens) - Add esbuild build pipeline (scripts/build.js, package.json) - Add composer dependencies (vendor/) - Add frontend src directory (assets/js/src/index.jsx) - Add documentation files - Remove old/obsolete files (class-settings.php, old CSS) This commits all pending changes from previous refactoring efforts.
1.4 KiB
1.4 KiB
Tokenizer
A small library for converting tokenized PHP source code into XML.
Installation
You can add this library as a local, per-project dependency to your project using Composer:
composer require theseer/tokenizer
If you only need this library during development, for instance to run your project's test suite, then you should add it as a development-time dependency:
composer require --dev theseer/tokenizer
Usage examples
$tokenizer = new TheSeer\Tokenizer\Tokenizer();
$tokens = $tokenizer->parse(file_get_contents(__DIR__ . '/src/XMLSerializer.php'));
$serializer = new TheSeer\Tokenizer\XMLSerializer();
$xml = $serializer->toXML($tokens);
echo $xml;
The generated XML structure looks something like this:
<?xml version="1.0"?>
<source xmlns="https://github.com/theseer/tokenizer">
<line no="1">
<token name="T_OPEN_TAG"><?php </token>
<token name="T_DECLARE">declare</token>
<token name="T_OPEN_BRACKET">(</token>
<token name="T_STRING">strict_types</token>
<token name="T_WHITESPACE"> </token>
<token name="T_EQUAL">=</token>
<token name="T_WHITESPACE"> </token>
<token name="T_LNUMBER">1</token>
<token name="T_CLOSE_BRACKET">)</token>
<token name="T_SEMICOLON">;</token>
</line>
</source>