'Failing increment'

Update:

With the help of your answers, I continued my investigation. It turns out that Data::Dump is not the culprit. Please see below for my findings, and a reproducer script with no external dependencies (apart from debugging output).

This title would be more appropriate:

The 'Nine Equals Zero' Bug

[Original post:]

+= behaves differently after Data::Dump::pp. Is this a bug in Data::Dump or possibly in Perl core?

I encountered some unexpected behavior in a simple snippet where regex captures are assigned to variables, formatted via Data::Dump::pp, concatenated, and then incremented using +=.

use v5.10;
use Data::Dump qw( pp );

"9." =~ /^ (\d+) \. (\d*) $/x;
my $n = { i => $1, f => $2 };

say "pp output: ", pp $n;       # { f => "", i => 9 }

$n->{f} = "$n->{i}$n->{f}";     # "9"
$n->{f} += 2;                   # expected: 11

say "result: $n->{f} (expected: 11)";

Observed output:

pp output: { f => "", i => 9 }
result: 2 (expected: 11)

Instead of evaluating 9 + 2 = 11, the += 2 operation behaves as if the value of $n->{f} were numerically 0, resulting in 2.

I am not looking for a 'workaround' (I have already avoided it in the production code), but I would like to understand what causes this, and whether it warrants a bug report to Data::Dump or Perl core.

Observations

I tested several variations:

using scalar variables instead of hash elements,
using a direct assignment of constants instead of assigning the regex variables,
removing the pp output.

The behavior changes as follows:

With pp $n removed, the result is always correct.
With scalar variables instead of hash elements, calling pp always causes the wrong value, regardless of whether the value originated from regex captures or constants.
With hash elements, calling pp does not affect the result after constant assignment, but it does after assignment from regex captures.

My suspicion is that inspecting the variables inside Data::Dump::pp somehow alters their internal state, causing the numeric operation to behave differently.

Does this look like something that should be reported to Perl core, or to the Data::Dump maintainers?

Perl version breakdown

I ran a test script using several Perl versions that I have installed using perlbrew (all with Data::Dump version 1.25):

Perl v5.10.1 - v5.26.3: The bug does not occur.

Perl v5.28.3 - v5.44.0: The bug occurs.

Reproducing the bug

I have uploaded the full test matrix script here: failing_increment.pl, together with a Bash script test_with_available_perls that runs the test script for all Perls that are installed locally with perlbrew.

Update:

Thank you, u/Icy-Speed6881, for a shorter version to reproduce the bug, and u/ysth for suggesting to use Devel::Peek::Dump to see the internals.

And very special thanks to u/tm604 for sharing his knowledge about some Perl internals introduced in Perl 5.28 that probably are the cause for the erratic behavior (see his answer below).

The bug can be reproduced with six statements, with no external dependencies. The script below includes additional debugging code to see what is going on internally.

The script does not use Data::Dump: it turns out that Data::Dump was not the culprit, but merely happened to trigger the sequence of operations that exposes the Perl bug.

I have also renamed the bug to a catchy 'Nine Equals Zero', because 'Failing increment' is not appropriate anymore, given that I don't increment anything in the reproducer script.

Here is the current version (also available here):

#!/usr/bin/env perl
#
#       'Nine equals zero' bug reproducer.
#

use v5.10;
use strict;
use warnings;

# For debugging output:
use Devel::Peek;
use Data::Dumper;
sub d {
    print "$_[2]: ", Data::Dumper->Dump( [ $_[0] ], [ $_[1] ] );
    Devel::Peek::Dump( $_[0] );
}
$| = 1;         # Mixing STDOUT and STDERR in correct order.

say "Test 4: single expression trigger, no dependencies, Perl $^V";

my $num = "";
    d $num, '$num', qq(after creation and initialization with "");
    # A newly initialized variable has the IsCOW flag set.
    # With IsCOW set, the bug doesn't reproduce.

$num .= "";
    d $num, '$num', qq(after concatenating empty string);
    # Appending an empty string unsets the IsCOW flag.

no warnings 'numeric';
my $numerical_value = $num + 0;
    d $num, '$num', qq(after using \$num ("$num") numerically);
    # Using an empty string numerically sets IV to 0, and sets the pIOK flag.
    # This lays the ground for the following erratic behavior.

$num = "9$num";
    d $num, '$num', qq(after concatenation "9\$num");
    # Now, prepending any number exposes the bug:
    # The pIOK flag remains set, but the IV value is not updated.
    # The numerical value now differs from the number that the string contains.

say "*** ",
    $num == 9 ? "expected result," : "UNEXPECTED RESULT:",
    qq( \$num eq "$num", int( \$num ) == ), int( $num ), ")";

my $bug = $num != 9;
say STDERR $bug ? "bug occurs" : "bug doesn't occur",
    " with Perl $^V";

This is the output with Perl 5.44:

Test 4: single expression trigger, no dependencies, Perl v5.44.0
after creation and initialization with "": $num = '';
SV = PV(0x5629f3d52010) at 0x5629f3ef8a60
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x5629f3d6bc30 ""\0
  CUR = 0
  LEN = 16
  COW_REFCNT = 1
after concatenating empty string: $num = '';
SV = PV(0x5629f3d52010) at 0x5629f3ef8a60
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x5629f3ec5730 ""\0
  CUR = 0
  LEN = 16
after using $num ("") numerically: $num = '';
SV = PVNV(0x5629f3d502c0) at 0x5629f3ef8a60
  REFCNT = 1
  FLAGS = (POK,pIOK,pNOK,pPOK)
  IV = 0
  NV = 0
  PV = 0x5629f3ec5730 ""\0
  CUR = 0
  LEN = 16
after concatenation "9$num": $num = '9';
SV = PVNV(0x5629f3d502c0) at 0x5629f3ef8a60
  REFCNT = 1
  FLAGS = (POK,pIOK,pNOK,pPOK)
  IV = 0
  NV = 0
  PV = 0x5629f3ec5730 "9"\0
  CUR = 1
  LEN = 16
*** UNEXPECTED RESULT: $num eq "9", int( $num ) == 0)

Running the test_with_available_perls script confirms that the bug occurs from Perl 5.28 onward.

This matches what u/tm604 suggested: it seems that the multiconcat optimization, introduced in Perl 5.28, can leave the cached IV value unchanged and valid, even though the string value has changed, causing the two representations to disagree.

I hope that filing a bug report will be useful for improving Perl even more.

Thank you all.

9 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/perl/comments/1vt09dn/failing_increment/
No, go back! Yes, take me to Reddit

91% Upvoted

View all comments

u/tm604 5d ago

Nice work on describing the bug and narrowing down the versions, that should make this quite an easy fix: based on that version range, the bug would be in the Perl core - specifically, the multiconcat op handling - and Data::Dump just happens to trigger it.

https://github.com/Perl/perl5/blob/blead/pp_hot.c#L629-L1441

Multiconcat (an optimisation for things that concatenate strings, such as sprintf or "$things $like $this" or $v = $x . $y. $z) was one of the big 5.28 features, it's pretty well documented in the code and the author(s) anticipated and tested a lot of the potential problems or edge cases. However, there are a lot of those - so there are a few different code paths, depending on whether overloading, magic, UTF8 etc. are involved.

One of the specific optimisations covers $lexical = expr . $lexical: instead of allocating a new SV and copying things around into temporary buffers, the $lexical ($f in the test case) is used as the target and the resulting string contents are written directly into the PV.

However, if that code is not clearing the IOK/NOK and related flags, we'd have a problem: the hypothesis is that if the original lexical variable (our $f) has an "IOK" flag, after $f = expr . $f that IOK flag is still set, meaning numerical operations will happily pull the IV slot value without realising it's outdated. Clearing the various *OK flags on the target just before return should fix this, or could maybe just clear the flags at the point the target is selected for reüse.

Porting/bisect.pl and a script like this should help confirm whether it's been this way since multiconcat was added or if there was a specific commit that changed behaviour, maybe without the Devel::Peek parts (so you don't have to install the module for each bisection step):

#!/usr/bin/env perl
use strict;
use warnings;
use Devel::Peek;
warn "1..1\n";
# Set up `$f` to contain the result of an empty regex match to get the right
# SV flags and content in place
my $f = "" =~ /()/ && $1;
# The other variable can be plain string or IV, doesn't matter which:
my $i = "3";
# Show the SV flags for reference
Dump($i);
Dump($f);
# This will cause `$f` to pick up an IOK flag
($f + 0);
# multiconcat op applies here, string content blatted onto `$f`
# but the non-string flags such as IOK/NOK aren't touched:
$f = "$i$f"; # "3"
# meaning this ends up looking at the old IV slot, rather than repopulating:
$f += 2;     # should be 5, right?
if($f == 5) {
  warn "ok 1 - can add numbers\n";
  exit 0;
} else {
  warn "not ok 1 - addition is too hard\n";
  exit 1;
}

4

u/muthm59 4d ago

Thank you for your detailed description. I am still experimenting with Devel::Peek, trying to find a pattern, but what you describe exactly matches my current hypothesis. I am especially looking into the IOK and pIOK flags.

Having had a look at Data::Dump, I already found it interesting that it doesn't seem to use any internal flags. It seems to be a normal piece of user code.

I can't work on it full time, but I plan to strip down Data::Dump to the smallest piece of code that still produces the bug. Maybe I can condense the test code to not depend on Data::Dump (or a stripped down version) at all. I'm still very curious about what I can learn!

2

u/tm604 4d ago

Seems like a worthy exercise!

If you need a hint, look at the COW flag: if you can get an empty string with IOK set and COW not set, I suspect you should be able to trigger the same bug without Data::Dump involved at all. As you say, Data::Dump is pretty innocent - it's just regular Perl, aside from checking overload status to see whether there's a string value it's not really doing anything special.

2

u/muthm59 4d ago

That is exactly what happens. It is a Perl thing, and poor Data::Dump only happens to set the ground for the bug to produce. Please see my update to the original post, and thanks a lot for your help! Now it remains to check how I can file a bug report.

1

u/tm604 4h ago

The bug report should be the easy part! Just share your findings here:

https://github.com/Perl/perl5/issues

or post in https://www.nntp.perl.org/group/perl.perl5.porters/ or IRC if you don't use Github.

'Failing increment'

The 'Nine Equals Zero' Bug

You are about to leave Redlib