3 re::engine::Plugin - API to write custom regex engines
11 As of perl 5.9.5 it's possible to lexically replace perl's built-in
12 regular expression engine with your own (see L<perlreapi> and
13 L<perlpragma>). This module provides a glue interface to the relevant
14 parts of the perl C API enabling you to write an engine in Perl
15 instead of the C/XS interface provided by the core.
17 =head2 The gory details
19 Each regex in perl is compiled into an internal C<REGEXP> structure
20 (see L<perlreapi|perlreapi/The REGEXP structure>), this can happen
21 either during compile time in the case of patterns in the format
22 C</pattern/> or runtime for C<qr//> patterns, or something inbetween
23 depending on variable interpolation etc.
25 When this module is loaded into a scope it inserts a hook into
26 C<$^H{regcomp}> (as described in L<perlreapi> and L<perlpragma>) to
27 have each regexp constructed in its lexical scope handled by this
28 engine, but it differs from other engines in that it also inserts
29 other hooks into C<%^H> in the same scope that point to user-defined
30 subroutines to use during compilation, execution etc, these are
31 described in L</CALLBACKS> below.
33 The callbacks (e.g. L</comp>) then get called with a
34 L<re::engine::Plugin> object as their first argument. This object
35 provies access to perl's internal REGEXP struct in addition to its own
36 state (e.g. a L<stash|/stash>). The L<methods|/METHODS> on this object
37 allow for altering the C<REGEXP> struct's internal state, adding new
42 Callbacks are specified in the C<re::engine::Plugin> import list as
43 key-value pairs of names and subroutine references:
45 use re::engine::Plugin (
51 To write a custom engine which imports your functions into the
52 caller's scope use use the following snippet:
54 package re::engine::Example;
55 use re::engine::Plugin ();
59 # Sets the caller's $^H{regcomp} his %^H with our callbacks
60 re::engine::Plugin->import(
67 *unimport = \&re::engine::Plugin::unimport;
69 # Implementation of the engine
81 # return value discarded
84 Called when a regex is compiled by perl, this is always the first
85 callback to be called and may be called multiple times or not at all
86 depending on what perl sees fit at the time.
88 The first argument will be a freshly constructed C<re::engine::Plugin>
89 object (think of it as C<$self>) which you can interact with using the
90 L<methods|/METHODS> below, this object will be passed around the other
91 L<callbacks|/CALLBACKS> and L<methods|/METHODS> for the lifetime of
94 Calling C<die> or anything that uses it (such as C<carp>) here will
95 not be trapped by an C<eval> block that the pattern is in, i.e.
98 use re::engine::Plugin(
101 croak "Your pattern is invalid"
102 unless $rx->pattern =~ /pony/;
106 # Ignores the eval block
107 eval { /you die in C<eval>, you die for real/ };
109 This happens because the real subroutine call happens indirectly at
110 compile time and not in the scope of the C<eval> block. This is how
111 perl's own engine would behave in the same situation if given an
112 invalid pattern such as C</(/>.
117 use re::engine::Plugin(
121 # We always like ponies!
122 if ($str =~ /pony/) {
132 Called when a regex is being executed, i.e. when it's being matched
133 against something. The scalar being matched against the pattern is
134 available as the second argument (C<$str>) and through the L<str|/str>
135 method. The routine should return a true value if the match was
136 successful, and a false one if it wasn't.
138 This callback can also be specified on an individual basis with the
139 L</callbacks> method.
143 use re::engine::Plugin(
147 say 'matched ' ($ponies // 'no')
148 . ' pon' . ($ponies > 1 ? 'ies' : 'y');
154 Called when the regexp structure is freed by the perl interpreter.
155 Note that this happens pretty late in the destruction process, but
156 still before global destruction kicks in. The only argument this
157 callback receives is the C<re::engine::Plugin> object associated
158 with the regexp, and its return value is ignored.
160 This callback can also be specified on an individual basis with the
161 L</callbacks> method.
168 # in comp/exec/methods:
171 The last scalar to be matched against the L<pattern|/pattern> or
172 C<undef> if there hasn't been a match yet.
174 perl's own engine always stringifies the scalar being matched against
175 a given pattern, however a custom engine need not have such
176 restrictions. One could write a engine that matched a file handle
177 against a pattern or any other complex data structure.
181 The pattern that the engine was asked to compile, this can be either a
182 classic Perl pattern with modifiers like C</pat/ix> or C<qr/pat/ix> or
183 an arbitary scalar. The latter allows for passing anything that
184 doesn't fit in a string and five L<modifier|/mod> characters, such as
185 hashrefs, objects, etc.
190 say "has /ix" if %mod =~ 'i' and %mod =~ 'x';
192 A key-value pair list of the modifiers the pattern was compiled with.
193 The keys will zero or more of C<imsxp> and the values will be true
194 values (so that you don't have to write C<exists>).
196 You don't get to know if the C<eogc> modifiers were attached to the
197 pattern since these are internal to perl and shouldn't matter to
202 comp => sub { shift->stash( [ 1 .. 5 ) },
203 exec => sub { shift->stash }, # Get [ 1 .. 5 ]
205 Returns or sets a user defined stash that's passed around as part of
206 the C<$rx> object, useful for passing around all sorts of data between
207 the callback routines and methods.
212 my $minlen = $rx->minlen // "not set";
214 The minimum C<length> a string must be to match the pattern, perl will
215 use this internally during matching to check whether the stringified
216 form of the string (or other object) being matched is at least this
217 long, if not the regexp engine in effect (that means you!) will not be
220 The length specified will be used as a a byte length (using
221 L<SvPV|perlapi/SvPV>), not a character length.
229 # A dumb regexp engine that just tests string equality
230 use re::engine::Plugin comp => sub {
233 my $pat = $re->pattern;
243 Takes a list of key-value pairs of names and subroutines, and replace the
244 callback currently attached to the regular expression for the type given as
245 the key by the code reference passed as the corresponding value.
247 The only valid keys are currently C<exec> and C<free>. See L</exec> and
248 L</free> for more details about these callbacks.
254 my ($re, $paren) = @_;
259 my ($re, $paren, $rhs) = @_;
261 # return value discarded
264 my ($re, $paren) = @_;
270 Takes a list of key-value pairs of names and subroutines that
271 implement numbered capture variables. C<FETCH> will be called on value
272 retrieval (C<say $1>), C<STORE> on assignment (C<$1 = "ook">) and
273 C<LENGTH> on C<length $1>.
275 The second paramater of each routine is the paren number being
276 requested/stored, the following mapping applies for those numbers:
278 -2 => $` or ${^PREMATCH}
279 -1 => $' or ${^POSTMATCH}
284 Assignment to capture variables makes it possible to implement
285 something like Perl 6 C<:rw> semantics, and since it's possible to
286 make the capture variables return any scalar instead of just a string
287 it becomes possible to implement Perl 6 match object semantics (to
290 =head2 named_captures
294 perl internals still needs to be changed to support this but when it's
295 done it'll allow the binding of C<%+> and C<%-> and support the
296 L<Tie::Hash> methods FETCH, STORE, DELETE, CLEAR, EXISTS, FIRSTKEY,
301 =head2 C<REP_THREADSAFE>
303 True iff the module could have been built with thread-safety features
306 =head2 C<REP_FORKSAFE>
308 True iff this module could have been built with fork-safety features
309 enabled. This will always be true except on Windows where it's false
310 for perl 5.10.0 and below.
314 The only way to untaint an existing variable in Perl is to use it as a
315 hash key or referencing subpatterns from a regular expression match
316 (see L<perlsec|perlsec/Laundering and Detecting Tainted Data>), the
317 latter only works in perl's regex engine because it explicitly
318 untaints capture variables which a custom engine will also need to do
319 if it wants its capture variables to be untanted.
321 There are basically two ways to go about this, the first and obvious
322 one is to make use of Perl'l lexical scoping which enables the use of
323 its built-in regex engine in the scope of the overriding engine's
326 use re::engine::Plugin (
328 my ($re, $str) = @_; # $str is tainted
332 my ($re, $paren) = @_;
334 # This is perl's engine doing the match
337 # $1 has been untainted
344 The second is to use something like L<Taint::Util> which flips the
345 taint flag on the scalar without invoking the perl's regex engine:
348 use re::engine::Plugin (
350 my ($re, $str) = @_; # $str is tainted
354 my ($re, $paren) = @_;
356 # Copy $str and untaint the copy
357 untaint(my $ret = $str);
359 # Return the untainted value
366 In either case a regex engine using perl's L<regex api|perlapi> or
367 this module is responsible for how and if it untaints its variables.
371 L<perlreapi>, L<Taint::Util>
373 =head1 TODO & CAVEATS
381 Engines implemented with this module don't support C<s///> and C<split
382 //>, the appropriate parts of the C<REGEXP> struct need to be wrapped
387 Still not a complete wrapper for L<perlreapi> in other ways, needs
388 methods for some C<REGEXP> struct members, some callbacks aren't
393 Support overloading operations on the C<qr//> object, this allow
394 control over the of C<qr//> objects in a manner that isn't limited by
395 C<wrapped>/C<wraplen>.
399 '@{}' => sub { ... },
405 Support the dispatch of arbitary methods from the re::engine::Plugin
406 qr// object to user defined subroutines via AUTOLOAD;
408 package re::engine::Plugin;
412 my ($name) = $AUTOLOAD =~ /.*::(.*?)/;
413 my $cv = getmeth($name); # or something like that
417 package re::engine::SomeEngine;
423 $re->add_method( # or something like that
425 my ($re, @arg) = @_; # re::engine::Plugin, 1..5
431 use re::engine::SomeEngine;
435 $re->foshizzle(1..5);
439 Implement the dupe callback, test this on a threaded perl (and learn
440 how to use threads and how they break the current model).
444 Allow the user to specify ->offs either as an array or a packed
445 string. Can pack() even pack I32? Only IV? int?
449 Add tests that check for different behavior when curpm is and is not
454 Add tests that check the refcount of the stash and other things I'm
455 mucking with, run valgrind and make sure everything is destroyed when
460 Run the debugger on the testsuite and find cases when the intuit and
461 checkstr callbacks are called. Write wrappers around them and add
471 This module may happen to build with a C++ compiler as well, but don't rely on it, as no guarantee is made in this regard.
473 L<XSLoader> (standard since perl 5.6.0).
477 Please report any bugs that aren't already listed at
478 L<http://rt.cpan.org/Dist/Display.html?Queue=re-engine-Plugin> to
479 L<http://rt.cpan.org/Public/Bug/Report.html?Queue=re-engine-Plugin>
483 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason C<< <avar at cpan.org> >>
485 Vincent Pit C<< <perl at profvince.com> >>
489 Copyright 2007,2008 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
491 Copyright 2009,2010,2011,2013,2014,2015 Vincent Pit.
493 This program is free software; you can redistribute it and/or modify it
494 under the same terms as Perl itself.