Skip to content

Revcheck library and QA tools for translation synchronization  #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
Jun 14, 2024
Merged

Revcheck library and QA tools for translation synchronization  #111

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
9120926
First part of revcheck algorithm.
Jul 11, 2023
0bf8620
Hash and timestamp details for source files.
Jul 13, 2023
2eb3e3c
Revtag parser.
Jul 13, 2023
7b74f59
Rename for future use.
Jul 14, 2023
9fc530c
Implementation of revcheck algorithm.
Jul 14, 2023
7c5cc0c
Export some calculated status to easily use on other tools.
Jul 16, 2023
a5c212f
qaxml.a.check.php, tool to check tag-attribute-values.
Jul 17, 2023
ecb3f1e
Print diff in conventional order.
Jul 27, 2023
117660a
qaxml.a -- Documentation and small fixes.
Aug 2, 2023
6cf1423
Initial support for [skip-revcheck].
Aug 3, 2023
03b9760
[skip-revcheck] compatibility.
Aug 3, 2023
ea00f5e
Compact output.
Aug 3, 2023
7058ea8
Script to check entities usage in translatoins.
Aug 4, 2023
1f3c5f1
qaxml.p and qaxml.t tools.
Aug 4, 2023
e3a7609
Rename compare hashes to head and diff.
Aug 5, 2023
f3e45aa
Tools documentation.
Aug 5, 2023
45d1126
Merge branch 'php:master' into master
alfsb Aug 5, 2023
997af1f
Ignore order in comparation.
Aug 5, 2023
e07946b
Simplify output.
Aug 5, 2023
6d4a3fb
Small fixes.
Aug 7, 2023
66d9630
Simple and complex output.
Aug 7, 2023
40edf10
`bookinfo.xml` is expectd to tag differ.
Aug 7, 2023
7cc43f5
Case insentivity on `<type>`.
Aug 8, 2023
84818f6
Consistent output volunteer myself for revisions.
Aug 11, 2023
817fa07
Consistent output volunteer myself for revisions.
Aug 11, 2023
a933c9e
Merge branch 'php:master' into master
alfsb Aug 12, 2023
0cd98f5
Missing ;
Aug 12, 2023
afebb28
Merge branch 'php:master' into master
alfsb Aug 13, 2023
918bfc6
Fix revtag capture and error reporting.
Aug 13, 2023
60c9cae
Organize require for all files in `lib/`
Aug 14, 2023
cb1eb76
New detail output for `qaxml.t.php`.
Aug 14, 2023
bd4846d
New detail output for `qaxml.t.php`.
Aug 14, 2023
efbefa7
Detect and ignore expected outputs.
Aug 15, 2023
258bc50
New `qarvt.php` to validate revtag expected format.
Aug 17, 2023
34f2e3d
Offer ignore comment only if checking tag's contents.
Aug 18, 2023
aa9b422
Save ignore locally, outside of global XML.
Aug 18, 2023
2ab0f2f
Enhance ignore marking to consider arguments and intenal state.
Aug 18, 2023
b1d1205
Ignore usage in all cases.
Aug 18, 2023
2b3d81e
Merge branch 'php:master' into master
alfsb Aug 18, 2023
1292b1f
Generic algorithm to detect count and contents mismatch.
Aug 22, 2023
cda8129
Count filtered.
Aug 22, 2023
e8dcd15
Generalize case insensitivity on basic types
Aug 22, 2023
ae34a2e
Configurable ignore output.
Aug 22, 2023
99d9e30
Fix wrong hash on unused message.
Aug 22, 2023
1c1c59b
Simplify output.
Aug 23, 2023
43624a2
Add ignore capacity on qaxml.e.php.
Aug 24, 2023
2e3585a
Convert qaxml.t.php to use OutputIgnoreBuffer.
Aug 24, 2023
beec19b
Fix instable output on qaxml.t.php.
Aug 25, 2023
467d573
Merge branch 'php:master' into master
alfsb Oct 9, 2023
da1a6d9
Merge branch 'php:master' into master
alfsb Oct 17, 2023
92964f2
Merge branch 'php:master' into master
alfsb Nov 2, 2023
2614f14
Merge branch 'php:master' into master
alfsb Nov 21, 2023
ed29a72
License code to PHP License
Dec 14, 2023
3a9dd37
Merge branch 'php:master' into master
alfsb Dec 14, 2023
0861659
Fix dynamic property creation into temporary
Dec 15, 2023
ed0a911
Usage and help info
Dec 15, 2023
07eee25
Documentation fixes
Dec 15, 2023
821023f
Pass by reference is not necessary with classes
Dec 15, 2023
bfa2f52
Encapsulate private property
Dec 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions scripts/translation/lib/RevcheckRun.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ class RevcheckRun
public RevcheckFileList $targetFiles;

// Separated lists
public array $filesOk;
public array $filesOld;
public array $filesRevtagProblem;
public array $filesUntranslated;
public array $filesNotInEn;
public array $filesWip;
public array $qaList;
public array $filesOk = [];
public array $filesOld = [];
public array $filesRevtagProblem = [];
public array $filesUntranslated = [];
public array $filesNotInEn = [];
public array $filesWip = [];
public array $qaList = [];

function __construct( string $sourceDir , string $targetDir , bool $writeResults = true )
{
Expand Down
40 changes: 21 additions & 19 deletions scripts/translation/qaxml.a.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,33 +30,35 @@
if ( implode( "\n" , $s ) == implode( "\n" , $t ) )
continue;

$intersect = array_intersect( $s, $t );
$onlySource = array_diff( $s , $intersect );
$onlyTarget = array_diff( $t , $intersect );
$header = true;
$match = array();

print "qaxml.a: {$target}\n\n";
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

foreach( $onlyTarget as $only )
print "- {$only}\n";
foreach( $onlySource as $only )
print "+ {$only}\n";
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
foreach( $match as $k => $v )
{
$s = array_count_values( $s );
$t = array_count_values( $t );
foreach ($s as $key => $countSource )
if ( $v[0] == $v[1] )
continue;

if ( $header )
{
$countTarget = $t[$key];
$countDiff = $countSource - $countTarget;
if ( $countDiff > 0 )
print "* {$key} +{$countDiff}\n";
if ( $countDiff < 0 )
print "* {$key} {$countDiff}\n";
print "qaxml.a: {$target}\n\n";
$header = false;
}

print "* {$k} -{$v[1]} +{$v[0]}\n";
}

print "\n";
if ( ! $header )
print "\n";
}

function extractTriple( array $list )
Expand Down
40 changes: 21 additions & 19 deletions scripts/translation/qaxml.e.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,31 +24,33 @@
if ( implode( "\n" , $s ) == implode( "\n" , $t ) )
continue;

$intersect = array_intersect( $s, $t );
$onlySource = array_diff( $s , $intersect );
$onlyTarget = array_diff( $t , $intersect );
$header = true;
$match = array();

print "qaxml.e: {$target}\n\n";
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

foreach( $onlyTarget as $only )
print "- {$only}\n";
foreach( $onlySource as $only )
print "+ {$only}\n";
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
foreach( $match as $k => $v )
{
$s = array_count_values( $s );
$t = array_count_values( $t );
foreach ($s as $key => $countSource )
if ( $v[0] == $v[1] )
continue;

if ( $header )
{
$countTarget = $t[$key];
$countDiff = $countSource - $countTarget;
if ( $countDiff > 0 )
print "* {$key} +{$countDiff}\n";
if ( $countDiff < 0 )
print "* {$key} {$countDiff}\n";
print "qaxml.e: {$target}\n\n";
$header = false;
}

print "* &{$k}; -{$v[1]} +{$v[0]}\n";
}

print "\n";
if ( ! $header )
print "\n";
}
40 changes: 21 additions & 19 deletions scripts/translation/qaxml.p.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,33 +30,35 @@
if ( implode( "\n" , $s ) == implode( "\n" , $t ) )
continue;

$intersect = array_intersect( $s, $t );
$onlySource = array_diff( $s , $intersect );
$onlyTarget = array_diff( $t , $intersect );
$header = true;
$match = array();

print "qaxml.p: {$target}\n\n";
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

foreach( $onlyTarget as $only )
print "- {$only}\n";
foreach( $onlySource as $only )
print "+ {$only}\n";
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
foreach( $match as $k => $v )
{
$s = array_count_values( $s );
$t = array_count_values( $t );
foreach ($s as $key => $countSource )
if ( $v[0] == $v[1] )
continue;

if ( $header )
{
$countTarget = $t[$key];
$countDiff = $countSource - $countTarget;
if ( $countDiff > 0 )
print "* {$key} +{$countDiff}\n";
if ( $countDiff < 0 )
print "* {$key} {$countDiff}\n";
print "qaxml.p: {$target}\n\n";
$header = false;
}

print "* {$k} -{$v[1]} +{$v[0]}\n";
}

print "\n";
if ( ! $header )
print "\n";
}

function extractPiData( array $list )
Expand Down
122 changes: 51 additions & 71 deletions scripts/translation/qaxml.t.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,31 +79,23 @@
$s = extractTagsInnerText( $s , $tags );
$t = extractTagsInnerText( $t , $tags );

$intersect = array_intersect( $s, $t );
$onlySource = array_diff( $s , $intersect );
$onlyTarget = array_diff( $t , $intersect );
$match = array();

if ( count( $s ) == count( $t ) && count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
continue;
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

foreach( $onlyTarget as $only )
$output->push( "- {$only}\n" );
foreach( $onlySource as $only )
$output->push( "+ {$only}\n" );
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
foreach( $match as $k => $v )
{
$s = array_count_values( $s );
$t = array_count_values( $t );
foreach ($s as $key => $countSource )
{
$countTarget = $t[$key];
$countDiff = $countSource - $countTarget;
if ( $countDiff > 0 )
$output->push( "* {$key} +{$countDiff}\n" );
if ( $countDiff < 0 )
$output->push( "* {$key} {$countDiff}\n" );
}
if ( $v[0] == $v[1] )
continue;
$output->push( "* {$k} -{$v[1]} +{$v[0]}\n" );
}

$output->pushExtra( "\n" );
Expand All @@ -122,31 +114,23 @@
$s = extractTagsInnerXmls( $s , $tags );
$t = extractTagsInnerXmls( $t , $tags );

$intersect = array_intersect( $s, $t );
$onlySource = array_diff( $s , $intersect );
$onlyTarget = array_diff( $t , $intersect );
$match = array();

if ( count( $s ) == count( $t ) && count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
continue;
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

foreach( $onlyTarget as $only )
$output->push( "- {$only}\n" );
foreach( $onlySource as $only )
$output->push( "+ {$only}\n" );
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( count( $onlySource ) == 0 && count( $onlyTarget ) == 0 )
foreach( $match as $k => $v )
{
$s = array_count_values( $s );
$t = array_count_values( $t );
foreach ($s as $key => $countSource )
{
$countTarget = $t[$key];
$countDiff = $countSource - $countTarget;
if ( $countDiff > 0 )
$output->push( "* {$key} +{$countDiff}\n" );
if ( $countDiff < 0 )
$output->push( "* {$key} {$countDiff}\n" );
}
if ( $v[0] == $v[1] )
continue;
$output->push( "* {$k} -{$v[1]} +{$v[0]}\n" );
}

$output->pushExtra( "\n" );
Expand All @@ -162,24 +146,28 @@
$s = XmlUtil::listNodeType( $s , XML_ELEMENT_NODE );
$t = XmlUtil::listNodeType( $t , XML_ELEMENT_NODE );

$s = countTags( $s );
$t = countTags( $t );
$s = extractNodeName( $s );
$t = extractNodeName( $t );

equalizeKeys( $s , $t );
equalizeKeys( $t , $s );
$match = array();

foreach( $s as $tag => $sourceCount )
{
$targetCount = $t[$tag];
foreach( $s as $v )
$match[$v] = array( 0 , 0 );
foreach( $t as $v )
$match[$v] = array( 0 , 0 );

if ( $sourceCount != $targetCount )
{
$output->push( "* {$tag} -{$targetCount} +{$sourceCount}\n" );
foreach( $s as $v )
$match[$v][0] += 1;
foreach( $t as $v )
$match[$v][1] += 1;

if ( $showDetail )
printTagUsageDetail( $source , $target , $tag , $output );
}
foreach( $match as $k => $v )
{
if ( $v[0] == $v[1] )
continue;
$output->push( "* {$k} -{$v[1]} +{$v[0]}\n" );
}

$output->pushExtra( "\n" );
}

Expand Down Expand Up @@ -213,6 +201,14 @@
$output->print();
}

function extractNodeName( array $list )
{
$ret = array();
foreach( $list as $elem )
$ret[] = $elem->nodeName;
return $ret;
}

function extractTagsInnerText( array $nodes , array $tags )
{
$ret = array();
Expand Down Expand Up @@ -273,22 +269,6 @@ function extractTagsInnerXmls( array $nodes , array $tags )
return $ret;
}

function countTags( array $list )
{
$ret = array();
foreach( $list as $elem )
$ret[] = $elem->nodeName;
$ret = array_count_values( $ret );
return $ret;
}

function equalizeKeys( array $list , array & $other , mixed $value = 0 )
{
foreach( $list as $k => $v )
if ( ! isset( $other[$k] ) )
$other[$k] = $value;
}

function printTagUsageDetail( string $source , string $target , string $tag , OutputBufferHasher $output )
{
$output->push( "\n" );
Expand Down