Details

Reviewers

hach-que
epriestley

Group Reviewers

Blessed Reviewers

Commits

rARCbd191c2860e3: Fix issues in C# unit test engine

Summary

This fixes a few issues in the C# unit test engine. It fixes tests sitting in subdirectories not being tested correctly (the location of both the test assembly and the results file would be wrong). It also fixes a very strange issue where xUnit.NET seems to not output the resulting XML file when it executes; in this case we just retry running the test until the XML file appears after completion (and eventually it works).

Test Plan

Ran arc unit --everything and arc unit --everything --no-coverage and verified that it's all reliably working.

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

Is this writing to NFS or something? I don't love the:

while (true) {
  // check for ghosts
}

...construction. When it fails to write, does it exit with an error code? Anything on stderr?

src/unit/engine/XUnitTestEngine.php
318	If this is to avoid collisions, maybe this should have some `Filesystem::readRandomCharacters()` in it?
329–332	You can do this without the branch by using `DIRECTORY_SEPARATOR`.
381–385	uhhhhhhh

hach-que added inline comments.Nov 17 2013, 11:00 PM

src/unit/engine/XUnitTestEngine.php
318	This was to ensure any previous .results files that didn't get cleaned up weren't impacting future tests. It was only really relevant while I was trying to diagnose the randomness of the files not appearing.

I don't think I can reproduce this, although I'm not sure why the "Website" tests are failing.

P993 arc_unit.txt

1	ubuntu@ip-10-174-254-230:~/Tychaia$ arc unit --everything --trace
2	libphutil loaded from '/home/ubuntu/libphutil/src'.
3	arcanist loaded from '/home/ubuntu/arcanist/src'.
4	Loading phutil library from '/home/ubuntu/Tychaia/Build/Arcanist'...
5	>>> [0] <exec> $ git rev-parse --show-cdup
6	<<< [0] <exec> 11,425 us
7	>>> [1] <exec> $ git rev-parse --verify HEAD^
8	<<< [1] <exec> 10,753 us
9	>>> [2] <exec> $ git rev-parse --abbrev-ref --symbolic-full-name '@{upstream}'
10	<<< [2] <exec> 10,380 us
11	>>> [3] <exec> $ git cat-file -t 'origin/master'
12	<<< [3] <exec> 10,327 us
13	>>> [4] <exec> $ git merge-base 'origin/master' HEAD
14	<<< [4] <exec> 10,692 us
15	>>> [5] <exec> $ git diff --no-ext-diff --no-textconv --raw '68b860137eee5af0f78bfdd38ac25768a93a45e1' --
16	<<< [5] <exec> 104,350 us
17	>>> [6] <exec> $ git diff --no-ext-diff --no-textconv --raw 'HEAD' --
18	>>> [7] <exec> $ git ls-files --others --exclude-standard
19	>>> [8] <exec> $ git ls-files -m
20	<<< [8] <exec> 41,625 us
21	<<< [7] <exec> 50,679 us
22	<<< [6] <exec> 132,014 us
23	>>> [9] <exec> $ which 'msbuild'
24	<<< [9] <exec> 9,803 us
25	>>> [10] <exec> $ which 'xbuild'
26	<<< [10] <exec> 9,449 us
27	>>> [11] <exec> $ which 'mono'
28	<<< [11] <exec> 9,458 us
29	>>> [12] <exec> $ which 'mono'
30	<<< [12] <exec> 9,476 us
31	Loading tests...
32	Test: Protogame/Protogame.Tests
33	Test: Protogame/ThirdParty/Dx/Dx.Runtime.Tests
34	Test: Tychaia.Game.Tests
35	Test: Tychaia.Globals.Tests
36	Test: Tychaia.ProceduralGeneration.Tests
37	Test: Tychaia.Runtime.Tests
38	Test: Tychaia.Tests
39	Test: Tychaia.Threading.Tests
40	Test: Tychaia.Website.Tests
41	>>> [13] <exec> $ /usr/bin/mono Protobuild.exe --resync 'Linux'
42	<<< [13] <exec> 10,877,519 us
43	>>> [14] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
44	<<< [14] <exec> 8,486,907 us
45	>>> [15] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
46	<<< [15] <exec> 4,123,009 us
47	>>> [16] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
48	<<< [16] <exec> 7,423,876 us
49	>>> [17] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
50	<<< [17] <exec> 1,517,177 us
51	>>> [18] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
52	<<< [18] <exec> 8,912,701 us
53	>>> [19] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
54	<<< [19] <exec> 9,533,975 us
55	>>> [20] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
56	<<< [20] <exec> 22,604,636 us
57	>>> [21] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
58	<<< [21] <exec> 5,266,287 us
59	>>> [22] <exec> $ xbuild '/p:SkipTestsOnBuild=True'
60	<<< [22] <exec> 1,814,386 us
61	PASS 10.9s (regenerate projects for Linux)
62	PASS 8.5s (build) Protogame/Protogame.Tests
63	PASS 12.6s (build) Protogame/ThirdParty/Dx/Dx.Runtime.Tests
64	PASS 20.0s (build) Tychaia.Game.Tests
65	PASS 21.6s (build) Tychaia.Globals.Tests
66	PASS 30.5s (build) Tychaia.ProceduralGeneration.Tests
67	PASS 40.0s (build) Tychaia.Runtime.Tests
68	PASS 1m02s (build) Tychaia.Tests
69	PASS 1m07s (build) Tychaia.Threading.Tests
70	FAIL (build) Tychaia.Website.Tests
71	XBuild Engine Version 2.10.8.1
72	Mono, Version 2.10.8.1
73	Copyright (C) Marek Sieradzki 2005-2008, Novell 2008-2011.
74
75	Build started 11/18/2013 12:08:14 AM.
76	__________________________________________________
77	Project "/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj" (default target(s)):
78	Target PrepareForBuild:
79	Configuration: Debug Platform: AnyCPU
80	Target ResolveProjectReferences:
81	Project "/home/ubuntu/Tychaia/Phabricator.Conduit/Phabricator.Conduit.Linux.csproj" (default target(s)):
82	Target PrepareForBuild:
83	Configuration: Debug Platform: AnyCPU
84	Target ResolveAssemblyReferences:
85	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Reference 'System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' not resolved
86	For searchpath /home/ubuntu/Tychaia/Libraries/Microsoft
87	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Web.DataVisualization' as a file, but the file does not exist
88	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Web.DataVisualization.exe' as a file, but the file does not exist
89	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Web.DataVisualization.dll' as a file, but the file does not exist
90	For searchpath {CandidateAssemblyFiles}
91	Warning: {CandidateAssemblyFiles} not supported currently
92	For searchpath {HintPathFromItem}
93	HintPath attribute not found
94	For searchpath {TargetFrameworkDirectory}
95	Considered target framework dir /usr/lib/mono/4.0, assembly named 'System.Web.DataVisualization' not found.
96	For searchpath {PkgConfig}
97	Considered System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, but could not find in any pkg-config files.
98	For searchpath {GAC}
99	Considered System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, but could not find in the GAC.
100	For searchpath {RawFileName}
101	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' as a file, but the file does not exist
102	For searchpath bin/Debug/
103	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Web.DataVisualization' as a file, but the file does not exist
104	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Web.DataVisualization.exe' as a file, but the file does not exist
105	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Web.DataVisualization.dll' as a file, but the file does not exist
106	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Reference 'System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' not resolved
107	For searchpath /home/ubuntu/Tychaia/Libraries/Microsoft
108	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Data.Entity' as a file, but the file does not exist
109	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Data.Entity.exe' as a file, but the file does not exist
110	Considered '/home/ubuntu/Tychaia/Libraries/Microsoft/System.Data.Entity.dll' as a file, but the file does not exist
111	For searchpath {CandidateAssemblyFiles}
112	Warning: {CandidateAssemblyFiles} not supported currently
113	For searchpath {HintPathFromItem}
114	HintPath attribute not found
115	For searchpath {TargetFrameworkDirectory}
116	Considered target framework dir /usr/lib/mono/4.0, assembly named 'System.Data.Entity' not found.
117	For searchpath {PkgConfig}
118	Considered System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, but could not find in any pkg-config files.
119	For searchpath {GAC}
120	Considered System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, but could not find in the GAC.
121	For searchpath {RawFileName}
122	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' as a file, but the file does not exist
123	For searchpath bin/Debug/
124	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Data.Entity' as a file, but the file does not exist
125	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Data.Entity.exe' as a file, but the file does not exist
126	Considered '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/System.Data.Entity.dll' as a file, but the file does not exist
127	Target GenerateSatelliteAssemblies:
128	No input files were specified for target GenerateSatelliteAssemblies, skipping.
129	Target CoreCompile:
130	Tool /usr/bin/dmcs execution started with arguments: /noconfig /debug:full /debug+ /optimize- /out:obj/Debug/Phabricator.Conduit.dll ConduitClient.cs ConduitException.cs Properties/AssemblyInfo.cs /target:library /unsafe+ /define:"DEBUG;PLATFORM_LINUX" /reference:/usr/lib/mono/4.0/System.dll /reference:/usr/lib/mono/4.0/System.Web.Extensions.dll /reference:/usr/lib/mono/4.0/Microsoft.CSharp.dll /reference:../Libraries/Microsoft/System.Web.WebPages.Deployment.dll /reference:../Libraries/Microsoft/System.Web.WebPages.dll /reference:../Libraries/Microsoft/System.Web.WebPages.Razor.dll /reference:../Libraries/Microsoft/System.Web.Helpers.dll /reference:../Libraries/Microsoft/System.Web.Mvc.dll /reference:../Libraries/Microsoft/System.Web.Razor.dll /reference:/usr/lib/mono/4.0/System.Core.dll /reference:/usr/lib/mono/4.0/mscorlib.dll /reference:/usr/lib/mono/4.0/System.Web.dll /reference:/usr/lib/mono/4.0/System.Configuration.dll /reference:/home/ubuntu/Tychaia/Libraries/Microsoft/Microsoft.Web.Infrastructure.dll /reference:/usr/lib/mono/4.0/System.Data.Linq.dll /reference:/usr/lib/mono/4.0/System.Drawing.dll /reference:/usr/lib/mono/4.0/System.Xml.dll /reference:/usr/lib/mono/4.0/System.Runtime.Caching.dll /reference:/usr/lib/mono/4.0/System.ComponentModel.DataAnnotations.dll /warn:4
131	Target DeployOutputFiles:
132	Copying file from '/home/ubuntu/Tychaia/Phabricator.Conduit/obj/Debug/Phabricator.Conduit.dll.mdb' to '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/Phabricator.Conduit.dll.mdb'
133	Copying file from '/home/ubuntu/Tychaia/Phabricator.Conduit/obj/Debug/Phabricator.Conduit.dll' to '/home/ubuntu/Tychaia/Phabricator.Conduit/bin/Debug/Phabricator.Conduit.dll'
134	Done building project "/home/ubuntu/Tychaia/Phabricator.Conduit/Phabricator.Conduit.Linux.csproj".
135	Project "/home/ubuntu/Tychaia/Tychaia.Globals/Tychaia.Globals.Linux.csproj" (default target(s)):
136	Target PrepareForBuild:
137	Configuration: Debug Platform: AnyCPU
138	Target ResolveAssemblyReferences:
139	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
140	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
141	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
142	Target GenerateSatelliteAssemblies:
143	No input files were specified for target GenerateSatelliteAssemblies, skipping.
144	Target CoreCompile:
145	Skipping target "CoreCompile" because its outputs are up-to-date.
146	Done building project "/home/ubuntu/Tychaia/Tychaia.Globals/Tychaia.Globals.Linux.csproj".
147	/home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: error : /home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: /usr/lib/mono/xbuild/Microsoft/VisualStudio/v10.0/WebApplications/Microsoft.WebApplication.targets: Project file could not be imported, it was being imported by /home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: Imported project: "/usr/lib/mono/xbuild/Microsoft/VisualStudio/v10.0/WebApplications/Microsoft.WebApplication.targets" does not exist.
148	Task "MSBuild" execution -- FAILED
149	Done building target "ResolveProjectReferences" in project "/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj".-- FAILED
150	Done building project "/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj".-- FAILED
151
152	Build FAILED.
153
154	Warnings:
155
156	/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj (default targets) ->
157	/usr/lib/mono/4.0/Microsoft.Common.targets (ResolveProjectReferences target) ->
158	/home/ubuntu/Tychaia/Phabricator.Conduit/Phabricator.Conduit.Linux.csproj (default targets) ->
159	/usr/lib/mono/4.0/Microsoft.Common.targets (ResolveAssemblyReferences target) ->
160
161	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Reference 'System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' not resolved
162	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Reference 'System.Data.Entity, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' not resolved
163
164	/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj (default targets) ->
165	/usr/lib/mono/4.0/Microsoft.Common.targets (ResolveProjectReferences target) ->
166	/home/ubuntu/Tychaia/Tychaia.Globals/Tychaia.Globals.Linux.csproj (default targets) ->
167	/usr/lib/mono/4.0/Microsoft.Common.targets (ResolveAssemblyReferences target) ->
168
169	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
170	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
171	/usr/lib/mono/4.0/Microsoft.Common.targets: warning : Found a conflict between : 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' and 'System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Using 'System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' reference.
172
173	Errors:
174
175	/home/ubuntu/Tychaia/Tychaia.Website.Tests/Tychaia.Website.Tests.Linux.csproj (default targets) ->
176	/usr/lib/mono/4.0/Microsoft.Common.targets (ResolveProjectReferences target) ->
177
178	/home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: error : /home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: /usr/lib/mono/xbuild/Microsoft/VisualStudio/v10.0/WebApplications/Microsoft.WebApplication.targets: Project file could not be imported, it was being imported by /home/ubuntu/Tychaia/Tychaia.Website/Tychaia.Website.Linux.csproj: Imported project: "/usr/lib/mono/xbuild/Microsoft/VisualStudio/v10.0/WebApplications/Microsoft.WebApplication.targets" does not exist.
179
180	5 Warning(s)
181	1 Error(s)
182
183	Time Elapsed 00:00:01.5647330
184
185	ubuntu@ip-10-174-254-230:~/Tychaia$

See http://code.redpointsoftware.com.au/w/troubleshooting/mono/v10_dir_missing/. Mono doesn't seem to ship with the v10.0 directory set up correctly, even though .NET under Windows has it set up fine.

Also you won't be hitting the randomly failing section since it's not getting past the build.

I did some more investigation into this and it seems to be caused by a segmentation fault in Mono. After checking their bug list, it looks like it's fixed in a very new version of Mono (it's not available in the package repositories yet).

I'm going to install a later version of Mono and verify that the issue is fixed. If it is, I'll remove the do-while loop and restore the file_exists check to output a BROKEN status (with an additional message letting users know about the issues with the particular Mono version).

I've reported the bug at https://bugzilla.xamarin.com/show_bug.cgi?id=16379.

Not finished with these changes yet; just diffing it so I can access it later on a different machine.

hach-que planned changes to this revision.Nov 21 2013, 9:51 PM

Tested this and it's working; still need to verify Windows.

Verified that this works on Windows machines as well. @epriestley can you review?

One minor inline, feel free to send a followup if you want.

src/unit/engine/XUnitTestEngine.php
114	This should probably be `preg_quote()`, this `str_replace()` looks ghetto.

Closed by commit rARCbd191c2860e3 (authored by @jamesr, committed by @epriestley).

hach-que added inline comments.Nov 22 2013, 10:35 PM

src/unit/engine/XUnitTestEngine.php
114	Actually the input string is a regular expression, but in C# you don't place `/` around the regex, so `/` on it's own is fine in C#. Under PHP, it needs to be escaped because `preg_*` uses `/` to designate options. This is mainly to bring it inline with the linter discovery rules, where the linter discovery rules are passed straight into C#, which then does the discovery logic (as opposed to this unit test engine, where the discovery logic is implemented in PHP).

Fix issues in C# unit test engine
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 17231

src/unit/engine/CSharpToolsTestEngine.php

src/unit/engine/XUnitTestEngine.php

Fix issues in C# unit test engineClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 17231

src/unit/engine/CSharpToolsTestEngine.php

src/unit/engine/XUnitTestEngine.php

Fix issues in C# unit test engine
ClosedPublic
Actions

Revision Contents
Changeset List