codingScars

Getting data from C++ Build Insights SDK

2020-05-01T00:00:00+00:00

Welcome back to the series!

In the previous posts we’ve explored ways to find how long a C++ build takes, installed some tools and found a way to create flame graphs out of MSBuild’s execution.

Join me this time as we explore a brand new SDK that lets us get better data in an easier manner!

C++ Build Insights SDK

Last november @KevinCadieuxMS shared a post over at Microsoft’s C++ Team Blog announcing a brand new tool called vcperf to get extensive data out of a C++ compilation. It required getting Visual Studio 16.4 as vcperf is part of the installation.

Last march they went further:

They released the C++ Build Insights SDK so we can use it ourselves.
They open sourced vcperf, which serves as an example tool using the SDK.

So first of all, let me thank Kevin and all of the team for providing us with these extra tools. They’re highly appreciated and useful!

And there I was, finishing the initial release of my MSBuildFlameGraph tool as they released this SDK that made my tool almost deprecated! So, I set myself on a quest to explore it and this is what I found.

Disclaimer: this post is based on the 1.0.0 version of the C++ Build Insights SDK. It’s expected it evolves over time and some of this info can become stale!

Getting started

Instead of writing an incomplete and outdated list of steps to get it working, let me point you to the official documentation.

There, you’ll get info on how to collect a trace using vcperf or how the SDK is structured. We’ll be taking a look at some of its points now.

Activities and events

Put simply, the basic piece of information the SDK reports is the Event. They have an id, a timestamp, a name and the real process/thread/processor they were emitted from. They can also have extra data depending on which event type they are.

When an Event is instant they call them SimpleEvent, while it’s called Activity if it takes time to complete (and also provides the timestamp when it finished).

You can check the full list of activities and events at the official docs.

Hierarchies

Because activities have a Duration and take some time to complete, there can be other activities and events that execute within them. This way, an Activity can have both Activity and SimpleEvent children. On the other hand, every SimpleEvent is instant and can’t have any children.

After reading the event table and not being able to make a mental map myself, I decided to build a directed graph to visualize them.

These are the parent-child relationships for activities (yellow-bordered are top-level ones):

These are the parent-child relationships with SimpleEvent included (gray-shaded):

If we were to check the event table from the docs, we’d see each one of the event types we’ve seen matches a class: FRONT_END_PASS becomes FrontEndPass, WHOLE_PROGRAM_OPTIMIZATION becomes WholeProgramOptimization, etc.

Let’s keep this graph in mind as we continue exploring.

Exploring the SDK via code

When you want to analyze a build you’ll usually perform these steps:

Execute vcperf /start SomeTraceName from an elevated command-line prompt.
Build your project (from Visual Studio or command-line, no matter which).
Execute vcperf /stopnoanalyze SomeTraceName OutputFile.etl.
Analyze OutputFile.etl with your tool (using the SDK).

Please note that, although you can record traces yourself with the SDK, I’ll keep using vcperf for that.

This is the basic code:

#include <CppBuildInsights.hpp>
// shorthand namespace version for convenience
namespace CppBI = Microsoft::Cpp::BuildInsights;

class SampleAnalyzer : public CppBI::IAnalyzer
{
  // [...]
};

int main(int argc, char** argv)
{
  SampleAnalyzer analyzer;
  auto group = CppBI::MakeStaticAnalyzerGroup(&analyzer);
  int analysisPasses = 1;

  CppBI::RESULT_CODE result = CppBI::Analyze("OutputFile.etl", analysisPasses, group);

  return result == CppBI::RESULT_CODE::RESULT_CODE_SUCCESS ? EXIT_SUCCESS
                                                           : EXIT_FAILURE;
}

That’s all of the setup you need to do to get working! Now we need to know how to get useful data from the trace.

Please note I’ll be using CppBI as an alias to Microsoft::Cpp::BuildInsights so code sections require as little horizontal scrolling as possible.

IAnalyzer

There are three main member functions you need to know from the IAnalyzer:

CppBI::AnalysisControl OnStartActivity(const CppBI::EventStack&);
CppBI::AnalysisControl OnStopActivity(const CppBI::EventStack&);
CppBI::AnalysisControl OnSimpleEvent(const CppBI::EventStack&);

The SDK calls these whenever an Event is read from the .etl file. Which one depends on the kind of Event that was reported (Activity or SimpleEvent).

So what’s that EventStack? Whenever an Event is emitted we also get its context via this EventStack. Let’s say we get OnStopActivity called. An example of the EventStack could be:

Function         <- top of stack
Thread
CodeGeneration
C2DLL
BackEndPass
Compiler
-----------------

Or say we get OnSimpleEvent called:

SymbolName       <- top of stack
C1DLL
FrontEndPass
Compiler
-----------------

But how do we retrieve different events within the stack?

Matchers

Let’s say we want to know all of the command-line options within our trace. We explore the official docs and find there’s a CommandLine event type. We’d add this method to our class:

void OnCommandLineEvent(const CppBI::SimpleEvents::CommandLine& event)
{
  std::cout << event.Value() << std::endl;
}

The SDK has a very clever way of helping you get the events you’re interested in within a hierarchy:

class SampleAnalyzer : public CppBI::IAnalyzer
{
public:
  CppBI::AnalysisControl OnSimpleEvent(const CppBI::EventStack& eventStack) override
  {
    CppBI::MatchEventStackInMemberFunction(eventStack,
                                           this,
                                           &SampleAnalyzer::OnCommandLineEvent);

    return CppBI::AnalysisControl::CONTINUE;
  }
};

That MatchEventStackInMemberFunction call there makes magic and filters those stacks which contain the event types in OnCommandLineEvent signature.

We now know all of the command-line options within the trace, but they’re all mixed up. Remember how Compiler and Linker activities were parents of CommandLine events? Wouldn’t it be awesome to tell them apart?

Let’s create these two methods this time:

void OnCompilerCommandLineEvent(const CppBI::Activities::Compiler& compiler,
                                const CppBI::SimpleEvents::CommandLine& event)
{
  std::cout << "Compiler: " << event.Value() << std::endl;
}

void OnLinkerCommandLineEvent(const CppBI::Activities::Linker& linker,
                              const CppBI::SimpleEvents::CommandLine& event)
{
  std::cout << "Linker: " << event.Value() << std::endl;
}

See how the signature now takes a CommandLine preceded by either Compiler or Linker?. Now let’s update our OnSimpleEvent method to use them both:

CppBI::AnalysisControl OnSimpleEvent(const CppBI::EventStack& eventStack) override
{
  CppBI::MatchEventStackInMemberFunction(eventStack,
                                         this,
                                         &SampleAnalyzer::OnCompilerCommandLineEvent);

  CppBI::MatchEventStackInMemberFunction(eventStack,
                                         this,
                                         &SampleAnalyzer::OnLinkerCommandLineEvent);

  return CppBI::AnalysisControl::CONTINUE;
}

Magically, MatchEventStackInMemberFunction will match those stacks that have a Compiler activity and then a CommandLine event apart from those with a Linker activity and then a CommandLine event!

You can get more info on matchers and examples on the official docs.

Getting more useful data with the SDK

Now we know how to deal with the SDK, let’s try to get some data out of our build, shall we?

In this case we’ll be using a default Unreal Engine project. Oh, and because it’s the compiler and linker the ones emitting the events we can get data from projects not supported in MSBuildFlameGraph!

Collecting a trace

Before we can start, we need a trace to work with! These were my steps:

Installed Visual Studio 15.9.22 (Visual Studio 2017).
Cloned vcperf from GitHub and built it.
Installed Unreal Engine 4.24.3, launched it and created the default First Person Shooter project with C++ code.
Launched an elevated command prompt and navigated to the previous vcperf output directory.
Executed vcperf /start UE4Project.
In Visual Studio 2017, Rebuild Development|Win64 our Unreal Engine solution.
Executed vcperf /stopnoanalyze UE4Project UE4Project.etl.

Now we’ve got an UE4Project.etl trace to analyze!

Oh, the computer I ran it on is a 8-year-old i5 laptop with no SSD and low RAM, so expect slow times here!

Listing slowest files to compile

Before we get some code, we should go back to the directed graph we built before:

See how both FRONT_END_PASS and BACK_END_PASS live within the COMPILER activity? After a quick check with the official docs we can see both classes inherit from CompilerPass. And that class has two interesting members:

InputSourcePath: path to the source file that gets compiled.
OutputObjectPath: path to the compiled output file.

Also, because they inherit from Activity we can check the time it took from start to stop via Duration.

Nice! So we’ve got all the info we need. Now let’s write some code:

class FileCompilationsAnalyzer : public CppBI::IAnalyzer
{
public:
  FileCompilationsAnalyzer() : CppBI::IAnalyzer(), m_fileCompilationsData() {}
  virtual ~FileCompilationsAnalyzer() {}

  CppBI::AnalysisControl OnStopActivity(const CppBI::EventStack& eventStack) override
  {
    bool processed = CppBI::MatchEventStackInMemberFunction(eventStack,
                       this,
                       &FileCompilationsAnalyzer::OnFrontEndPassCompleted) ||
                     CppBI::MatchEventStackInMemberFunction(eventStack,
                       this,
                       &FileCompilationsAnalyzer::OnBackEndPassCompleted);
    return CppBI::AnalysisControl::CONTINUE;
  }

private:
  struct FileCompilationData
  {
    std::chrono::nanoseconds frontEndDuration;
    std::chrono::nanoseconds backEndDuration;
  };
  std::unordered_map<std::wstring, FileCompilationData> m_fileCompilationsData;

  void OnFrontEndPassCompleted(const CppBI::Activities::FrontEndPass& frontEndPass)
  {
    auto result = m_fileCompilationsData.try_emplace(frontEndPass.InputSourcePath(),
                                                     FileCompilationData());
    result.first->second.frontEndDuration = frontEndPass.Duration();
  }

  void OnBackEndPassCompleted(const CppBI::Activities::BackEndPass& backEndPass)
  {
    auto result = m_fileCompilationsData.try_emplace(backEndPass.InputSourcePath(),
                                                     FileCompilationData());
    result.first->second.backEndDuration = backEndPass.Duration();
  }
};

And that’s all we need! We can now run our code, sort the data we’ve collected and dump it:

path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectCharacter.cpp
  Total time: 82.724s (front-end: 81.086s, back-end: 1.638s)
path\DefaultFPSCppProjectEditor\Development\Engine\SharedPCH.Engine.ShadowErrors.h
  Total time: 81.302s (front-end: 74.104s, back-end: 7.198s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectGameMode.cpp
  Total time: 75.965s (front-end: 74.640s, back-end: 1.325s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectProjectile.cpp
  Total time: 75.914s (front-end: 74.615s, back-end: 1.300s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectHUD.cpp
  Total time: 75.535s (front-end: 74.570s, back-end: 0.964s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectCharacter.gen.cpp
  Total time: 3.071s  (front-end: 2.131s,  back-end: 0.940s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectGameMode.gen.cpp
  Total time: 3.024s  (front-end: 1.875s,  back-end: 1.149s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectHUD.gen.cpp
  Total time: 2.671s  (front-end: 2.030s,  back-end: 0.641s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProjectProjectile.gen.cpp
  Total time: 2.358s  (front-end: 1.654s,  back-end: 0.704s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProject.init.gen.cpp
  Total time: 2.253s  (front-end: 1.424s,  back-end: 0.829s)
path\UE4Editor\Development\DefaultFPSCppProject\DefaultFPSCppProject.cpp
  Total time: 1.212s  (front-end: 0.824s,  back-end: 0.388s)

That’s it, now we know which files take longer to compile! And it looks like front-end times are way larger than back-end ones!

Important note: it looks like projects compiled with Visual Studio 2017 (tested with version 15.9.22) report their InputSourcePath as null (but their OutputObjectPath is correct). This seems to work without a problem if you compile your project with Visual Studio 2019 (tested with version 16.5.2). For the sake of this section I had ommited this issue until now.

Listing slowest files to get included

This time we want to query the SDK for included files. Remember: file inclusion gets performed by the front-end. After a bit of searching we come up with FrontEndFile.

This Activity is recursive, as a file can include some file that in turn includes some other file. However, we’re only interested in files that get included from other file. That means we want to capture a stack with two FrontEndFile events!

Let’s see some code:

class FileInclusionsAnalyzer : public CppBI::IAnalyzer
{
public:
  FileInclusionsAnalyzer() : CppBI::IAnalyzer(), m_fileInclusionTimes() {}
  virtual ~FileInclusionsAnalyzer() {}

  CppBI::AnalysisControl OnStopActivity(const CppBI::EventStack& eventStack) override
  {
    CppBI::MatchEventStackInMemberFunction(eventStack,
                                           this,
                                           &FileInclusionsAnalyzer::OnFileParsed);
    return CppBI::AnalysisControl::CONTINUE;
  }

private:
  typedef std::vector<std::chrono::nanoseconds> TTimeElapsedPerOccurrence;
  typedef std::unordered_map<std::string, TTimeElapsedPerOccurrence> TTimeElapsedPerInclusion;
  TTimeElapsedPerInclusion m_fileInclusionTimes;

  void OnFileParsed(const CppBI::Activities::FrontEndFile& fromFile,
                    const CppBI::Activities::FrontEndFile& includedFile)
  {
    auto result = m_fileInclusionTimes.try_emplace(includedFile.Path(),
                                                   TTimeElapsedPerOccurrence());
    result.first->second.push_back(includedFile.Duration());
  }
};

And that’s it! Now we have a list of the times it took to include each file (one file can be included from several files!). Why don’t we sort all inclusions by total inclusion time and dump the first ones?

project\path\source\defaultfpscppproject\defaultfpscppprojectcharacter.h
  Total inclusion time: 49.854s (included 3 times)
ue_4.24\engine\source\runtime\engine\classes\gameframework\character.h
  Total inclusion time: 48.960s (included 3 times)
project\path\source\defaultfpscppproject\defaultfpscppprojectgamemode.h
  Total inclusion time: 43.096s (included 2 times)
ue_4.24\engine\source\runtime\engine\classes\gameframework\gamemodebase.h
  Total inclusion time: 42.405s (included 2 times)
ue_4.24\engine\source\runtime\engine\classes\gameframework\rootmotionsource.h
  Total inclusion time: 40.776s (included 3 times)
project\path\intermediate\[...]\engine\sharedpch.engine.shadowerrors.h
  Total inclusion time: 40.732s (included 1 times)
ue_4.24\engine\source\runtime\engine\public\enginesharedpch.h
  Total inclusion time: 40.729s (included 1 times)
project\path\source\defaultfpscppproject\defaultfpscppprojecthud.h
  Total inclusion time: 39.853s (included 3 times)
ue_4.24\engine\source\runtime\engine\classes\gameframework\hud.h
  Total inclusion time: 39.142s (included 3 times)
ue_4.24\engine\source\runtime\engine\classes\engine\serverstatreplicator.h
  Total inclusion time: 28.980s (included 2 times)
[...]

Easy!

Generate a directed graph for file inclusions

With the previous analyzer we could also get the inclusion relationships between files, can’t we? This time I won’t add any code, but a couple of screenshots of the resulting inclusion graph (.dgml format):

Yeah, no joke. That’s what I can see in a laptop resolution without zooming in! Let’s zoom a bit into something, shall we?

I’ve zoomed into the file we saw is the slowest to compile. It also shows which files include it and which files get included by it. To cut the graph down I installed Visual Studio’s DgmlPowerTools extension, Ctrl+F with DefaultFPSCppProjectCharacter.h to select the node and clicked the Butterfly option.

Generate a flame graph

There are other useful metrics to get, like how long do functions take to compile (careful, SDK reports their names mangled!), how many files get included on each .h file or whether files compile in parallel at all. I’m sure you can come up with more! But this time, we want to visualize the build.

First of all, we have to go back to vcperf. When it analyzes a trace, it generates extra data to view within Windows Performance Analyzer (WPA). However, although it gives very useful insights, I find myself a bit clumsy while exploring it. Please refer to vcperf on GitHub to get a glimpse of how it looks.

Over a year ago, @aras_p wrote this blog post on how to get a flame graph out of Clang’s compilations. So, after all this time, why don’t we try to get this kind of trace from MSVC?

We’d have to create a new analyzer to feed into the SDK that records all hierarchies and sets useful names for each entry in the graph. This means taking file paths from FrontEndFile, function names from Function and the like. Be careful, though, as reported events live in the stack and will get out of scope if you want to keep pointers to them!

Let’s take a look at the raw result!

Well, not specially readable either, isn’t it? To the left of the image we can see ProcessId and ThreadId as reported by the SDK. And if we want to know what’s going on we have to scroll back and forth, trying to know which entry matches which parent. Or, what’s compiling in parallel? Found myself feeling as clumsy as when using WPA.

But, can we do any better?

Packing parent-children together

Let’s post-process the data and try to keep child entries as close as possible to their parents, being careful to leave the necessary space to fit them when we have entries running in parallel.

With a bit of work, we get this version:

As you can see, ProcessId and ThreadId aren’t the real ones anymore. However, they approach to a more logical distribution of what’s running where! Definitely easier to read for me!

Let’s zoom into DefaultFPSCppProjectGameMode.gen.cpp:

What can we see here? As part of the C1DLL activity we see that this .cpp file includes three other files. We also see that the CodeGeneration activity has all of the functions that get generated (they get generated in parallel, up to 4 because my computer has 4 logical cores).

But we can see something else: the Compiler activity only spans the FrontEndPass and BackEndPass of this file. And, if we go back to the full trace, all Compiler activities only deal with one file. This maps to having a lot of CL MSBuild tasks! Does this mean we aren’t compiling in parallel?

Well, apparently we are compiling in parallel we check the full trace. It looks UnrealHeaderTool is spawning as many cl.exe in parallel as files we want to compile, instead of letting cl.exe do the parallelization itself?

Getting template instantiation data

The last cool thing we’ll see requires using the open source version of vcperf as well as compiling with Visual Studio 16.4 (Visual Studio 2019). This version of vcperf reports some extra events than the one shipping with Visual Studio 2019.

This time, we’ll use @BruceDawson0xB’s project from this blog post, but updated to Visual Studio 2019. Let’s take a look:

Bruce wanted to make compilations slow and calculated Fibonacci in templates (with some tricks to prevent the compiler from reusing template instantiations). And this is the result!

Of course, these template instantiations can be seen in other projects! This time we’ll build the tool I’ve used for this post in Release|x64 and zoom into Main.cpp:

See that huge amount of small entries? Those are template instantiations! And they can be part of any file, so when you include some file you also get which templates get instantiated.

However, have you noticed the small BackEndPass to the right? It’s got no children, hasn’t it? That’s because we’ve built a Release version which has LTCG enabled! Let’s take a look at the Linker activity:

Here’s where the Function activities get compiled! In a previous graph we saw them as part of the BackEndPass activity, but it didn’t have LTCG enabled.

Also, it looks like Visual Studio 2019 also adds Thread activities between CodeGeneration and Function activities!

This exploration of the C++ Build Insights SDK helped me take more informed decisions when investigating C++ compile times and I’m sure I’ll be using it in the future.

Again, let me thank the team that made the SDK possible, and @KevinCadieuxMS in particular!

To write this post I built a tool that’s now available on GitHub for everyone to try!

Thanks for reading!

Improving C++ compile times using flame graphs

2020-03-30T00:00:00+00:00

In the last posts we’ve explored ways to know how long a C++ build takes and how to create flame graphs out of MSBuild’s execution.

This time we’ll check several examples and try make more informed decisions to improve our build times. We’ll see a lot of examples, so expect it to be an image-heavy post!

Understanding MSBuild flame graphs

First of all, we should learn how to read these flame graphs. This one shows a sample build of my (unreleased) GameBoy emulator side project. It’s a Visual Studio 2015 solution that mixes C++, C++/CLI and C# code:

In this graph we’ve got:

Several pid: these are the NodeId values as reported by MSBuild. Because NodeId starts at 1, we’ve used index 0 to tag the build itself. They don’t exactly represent processors, cores or threads (we’ll see an example with a ton of timelines at the end of the post).
Several MSBuild timeline: they represent different execution hierarchies. Very important notice: they are absolutely made up. MSBuild reports entries (Project, Target or Task) living in a NodeId, but some of them don’t share the same hierarchy! For these cases, I decided to separate them into different timelines. We talked about hierarchies in the previous post.

It’s a bit hard to explore hierarchies with screenshots, but let’s edit the image above to help us understand what’s going on:

These are the two top-level projects within the solution. In this case, both create an executable (one for the emulator itself, one for the test suite).

Did you notice they’re .metaproj projects? That means they’re created implicitly from the .sln file because they have dependencies (more info in the previous post). See how both of them have two MSBuild tasks? The leftmost task builds its dependencies while the rightmost build the project itself.

In the image, the uppermost group labeled 1 is the MainWindows.vcxproj.metaproj. Its dependencies get built within the same timeline and then builds MainWindows.vcxproj down at NodeId 2. This was kind of surprising for me at first (why not building it within the same timeline, similar to its dependencies?) but that’s how MSBuild reports it.

Meanwhile, Test.vcxproj.metaproj is building in the same NodeId but in a different timeline: they build in parallel because they don’t depend on each other. It also waits for its dependencies (both projects share the same ones), then builds Test.vcxproj.

And what about these dependencies? Let’s zoom in a bit into MainWindows.vcxproj.metaproj and annotate the graph:

Group 1 is MainWindows.vcxproj.metaproj. It starts in NodeId 1, timeline 0 and features two MSBuild tasks. The rightmost one builds MainWindows.vcxproj in NodeId 2, timeline 0 (I guess it’s because MSBuild scheduled Test.vcxproj.metaproj to NodeId 1, so this one got moved to a separate one).
Group 2 is GameBoy.vcxproj.metaproj. Everything it does lives in NodeId 1, timeline 0.
Group 3 is LogLibrary.vcxproj.metaproj and, surprisingly, starts in NodeId 1, timeline 0 but does all of its work in NodeId 2, timeline 0. No idea why this jump, that’s how MSBuild scheduled them.
Group 4 is LogWindowBridge.vcxproj. Although it has a dependency, there’s no .vcxproj.metaproj this time. I guess it’s because it depends on a C# project? It does all of its work at NodeId 2, timeline 0, but I’ve painted the group bigger to fit the next one.
Group 5 is the last dependency: LogWindowUI.csproj. It gets built in NodeId 1, timeline 1 (MSBuild reports it shares NodeId 1 with other entries but they aren’t part of the same hierarchy, that’s why it goes to timeline 1).

I really hope it made things a bit more clear.

We can ask MSBuild to limit parallel projects to 1 and thus flatten them:

Leftmost entries are the ones that build first while depth represents dependencies.

Adding more info via MSVC flags

In the previous post we explored some MSVC flags that added extra timing information to our build, so we should try using them!

/Bt+

This flag produces this kind of output:

1>  File.cpp
1>  time(Z:\path\to\compiler\c1xx.dll)=0.39427s < 1617765075910 - 1617766036302 > BB [Z:\path\to\File.cpp]
1>  time(Z:\path\to\compiler\c2.dll)=0.06475s < 1617766053824 - 1617766211553 > BB [Z:\path\to\File.cpp]

These messages represent three parts of a file compilation: the start, the end of the compiler’s front-end and the end of the compiler’s back-end. With this, we can create our own entries in the graph and represent each compilation!

Because these entries are absolutely made up, I’ve labeled its timeline as Post-processed timeline to separate them from the raw MSBuild data.

/Bt+ output, for this graph, looks like this:

File1.cpp
time(c1xx.dll) [...] [File1.cpp]
File2.cpp
time(c1xx.dll) [...] [File2.cpp]
[...]
time(c2.dll) [...] [File2.cpp]
time(c2.dll) [...] [File1.cpp]

This means it’s executing the front-end for each file and then the back-end (and in reverse order)! That means we’re not compiling files in parallel, what a mess!

Turns out, this is the default behavior when you create a new project in Visual Studio 2015. Let’s fix it by adding the /MP flag:

We also have to turn off /Gm because it’s incompatible with /MP:

Let’s now rebuild the solution and check again!

That’s better! Now we can see how there’s as many Post-processed timeline as logical cores. This is the default value, it can be tweaked as well.

Again, because these timelines are made up, it’s not set in stone which file gets compiled in which core. All we have are MSBuild messages saying their compilation started, front-end execution finished and back-end execution finished. With that, we’ve filled the gaps in the logical cores.

Here we hit one big issue: every message within the same Task (CL task in this case) share the same BuildEventContext and other data we could use to tell messages apart. Luckily, /Bt+ messages say which file they refer to so we can know which ones are related. We’ll come back to this issue later on, with /d1reportTime.

Let’s use another project now! This is what /Bt+ has to say when we build the solution from this blog post:

First of all: no project has dependencies. That’s why we don’t find any .vcxproj.metaproj entries. Thanks to this, and because we’ve built the full solution, each project lies into its own NodeId. Let’s explore each one:

CompileNonParallel.vcxproj: it’s doesn’t have /MP enabled (we only see one Post-processed timeline) and it defines groups of files with different compiler flags (it has several CL tasks).
CompileMoreParallel.vcxproj: it has /MP enabled but still defines groups of files with different compiler flags. That’s why we see parallel compilations in the CL tasks with more than one file, but still a lot of CL tasks.
CompileMostParallel.vcxproj: all files share the same compiler flags except the pre-compiled header, that’s why we see two CL tasks.

But, what does this different compiler flags have to do with having several CL tasks? That’s because MSBuild creates batches of files that can be compiled with the same flags and then spawn as many CL tasks to execute the compiler. When they’re done, a new batch begins.

In the case of pre-compiled headers, you must compile one file with the /Yc flag to create the PCH while other files must compile with the /Yu flag to use that file. Because they’ve got different flags, MSBuild creates two batches.

Bonus: files with the same name

Let’s suppose you’ve got a project with a number of files spread into folders. And let’s say two of the files you’ve got use the same name: File.cpp. When you compile the project you get this warning:

Two or more files with the name of File.cpp will produce outputs to the same location. This can lead to an incorrect build result. The files involved are src\Folder1\File.cpp, src\Folder2\File.cpp.

This is because the default project configuration creates all intermediate .obj files in the same folder. When it compiles the second File.cpp file, its .obj file overwrites the former one. If you queried Google and checked the first answer in Stack Overflow, you’d modify the project with this property:

<ObjectFileName>$(IntDir)/%(RelativeDir)/</ObjectFileName>

And this is its trace:

Yes, it built without issues but now we will have as many CL tasks as folders in our project! This is the wrong approach!

Alright, that was a synthetic test, but now imagine you have a processor with 8 or 16 logical cores and the same kind of project. If you’ve got only a handful of files per folder you’d be wasting most of those cores because batches won’t have as many files! You’re destroying parallelism!

You better fix your physical structure instead of doing this, or your build times will suffer. I once decreased the Rebuild time of a project by almost 2 minutes (2 minutes!) with just this change. It can get serious.

/time+

Let’s move onto a linker flag we talked about in the previous post:

That’s a Debug|x64 compilation. Not many info, really. What about a Release|x64 compilation?

By default, Release configurations enable /LTCG (Link Time Code Generation). It first loads all of the .obj files (the small entries to the left of Pass 1) and then generates more optimized code (LTCG CodeGen entry).

/d1reportTime

This is the last flag we’ll be adding to get extra data (check previous post for more info). And it comes with two issues:

It’s only available from Visual Studio 2017 version of MSVC: this means we need to upgrade the GameBoy emulator project we’ve been using to make it work. Because we were already using MSBuild 15, we can build both VS2015 and VS2017 C++ solutions with the same tool!
It outputs a huge amount of Message tasks for each file and we can’t relate messages to each other (i.e. their BuildEventContext and other identifiable data match, but they don’t say which file they refer to like /Bt+ does). Because files are getting compiled in parallel we’ll get mixed Message tasks! We need to break parallelism to be able to build a hierarchy.

So let’s first upgrade our project and then ensure we only compile one file at a time. Doing so isn’t a hack either, that’s what the /showIncludes MSVC flag does (it shows a warning saying so, although it doesn’t get mentioned in the official docs).

Let’s take an example now!

This is one of my test files: LoadAIntoAddressHL.cpp. As reported by /d1reportTime, the front-end executes three sections: Include Headers, Class Definitions and Function Definitions. Again, all these entries are made up and that’s why they are in a Post-processed timeline.

Bonus: /Bt+ and /d1reportTime timing flaw

Let’s take the combined output for /Bt+ and /d1reportTime for this file and remove a lot of the messages, let’s just keep timing ones:

[...]
LoadAIntoAddressHL.cpp
Include Headers:
  [...]
  Total: 0.591067s
Class Definitions:
  [...]
  Total: 0.178967s
Function Definitions:
  [...]
  Total: 0.511502s
time(Z:\path\to\compiler\c1xx.dll)=1.17362s < 2283035638380 - 2283038497179 > [Z:\path\to\LoadAIntoAddressHL.cpp]

Let’s sum /d1reportTime sections: 0.591067s + 0.178967s + 0.511502s = ‭1.281536‬s. However, /Bt+ reports the front-end took 1.17362s! Disaster!

If we were to use these times in the graph as they’re reported, the Function Definitions entry would overflow c1xx.dll. To fix it I decided to scale down each child of c1xx.dll proportionally so all of them fit. This was a compromise, though: their times aren’t real anymore but still give you an idea of their relative impact.

Adding pre-compiled headers to the mix

With /d1reportTime data at hand, let’s add this PCH to the project and check how it changes:

#pragma once

#include <vector>
#include <string>
#include <iomanip>
#include <sstream>

First of all, we can see the Include Headers section features a smaller number of files and hierarchies: the biggest ones we had came from the STL! Class Definitions also has less entries: we can even read some of them at the same zoom level! And what about Function Definitions? It’s got far less entries now!

No wonder why the time spent on this file went down from 1.174s to 0.161s! Why don’t we compare how the Test.vcxproj project build time changed with the PCH?

This is how it looked before the PCH:

Let’s add the PCH now:

See how the PCH has cut a lot of depth on each file? Also, the full project went down from 2m17s to 49s, and that’s with /MP turned off! What if we turned it on?

This is how it looked before the PCH:

Let’s add the PCH now:

It’s gone from 1m08s to 27s! Also, have you noticed how each c1xx.dll entry has shrunk in size?

To sum up: we can check build times thanks to these flame graphs but also visualize the impact of our changes!

Extra examples

To finish this post I’ve built a couple of projects from GitHub to see how their flame graphs look like. Let’s check them!

RapidJSON

This is the flame graph for RapidJSON when building the whole solution in Debug|x64 with /Bt+ and /time+ enabled. Be warned, it’s large:

With the info from this post, can you tell whether /MP is enabled for this build?

For reference, this is what Paralel Builds Monitor has to say of the build:

See how the maximum number of projects building in parallel is 4? That’s the number of logical cores in the machine I used to build it.

Google Test

This one is for Google Test, Debug|x64. While RapidJSON built 28 projects, this one builds 80! We’ll have to see it as a downscaled gif instead of a huge screenshot:

This series has focused on exploring and understanding MSBuild to create flame graphs we could view in Google Chrome’s trace viewer. We’ve learned about MSVC flags and other interesting tools as well. Thanks to them we can make more informed decisions when improving build times and be able to measure their impact.

I hope any of this info was useful for you, it definitely was for me.

Thank you for your time, dear reader!

Update #1: I’ve decided to release this tool at GitHub so anyone can have a try. Thanks for all the support!

Understanding MSBuild to create flame graphs

2020-03-27T00:00:00+00:00

I love side projects. You have a problem you want to solve, spend some time thinking about how you’d do it and then start working on a solution that mostly works. When you’ve calmed the itch that started it all, it becomes a product and you start losing interest on it. A new side projects pops in, the cycle repeats.

But this time I managed to finish one!

Over a year ago I started a side project to create a flame graph out of a MSBuild execution. It was inspired by @aras_p’s blog post. Thanks again for everything you’ve written in your blog!

This post serves as a summary of what I learned while developing this tool. I thought it would be a shorter post but then got out of control! We’ll have an extra post fiddling with compiler flags and project configurations to show a lot of examples!

Some of the examples in this post will feature my (unreleased) GameBoy emulator (yeah, one of those abandoned side projects) to illustrate some points. This way, we can see some real data and not just synthetic tests.

The background

When I started looking for ways to investigate build times I ended up with some useful MSVC flags and some Visual Studio options (check previous post in the series for more).

It basically looked like I could compile a solution, parse its output and start working from there. And it didn’t seem very crazy, people build tools around that! Just open Visual Studio’s Developer Command Prompt and type:

devenv.com Z:\path\to\solution.sln /Rebuild

I’ve used devenv.com instead of devenv.exe because it writes to console.

We’ll get something like:

3>  [...]
4>------ Rebuild All started: Project: GameBoy, Configuration: Debug x64 ------
4>  Cartridge.cpp
4>  [...]
========== Rebuild All: 6 succeeded, 0 failed, 0 skipped ==========

This is basically like building the solution in Visual Studio and copying the Output panel. However, that’s all you have. I guess it isn’t enough for us!

In the previous post I mentioned Visual Studio Extensions use an SDK that lets you hook to MSBuild events and perform your own logic. That means Visual Studio is using MSBuild under the hood, doesn’t it?

Let’s try to build our solution with MSBuild directly (again, within Visual Studio’s Developer Command Prompt):

msbuild.exe Z:\path\to\solution.sln /t:Rebuild

This will rebuild our solution and dump a lot of information to console. It looks very much like setting MSBuild project build output verbosity to Normal or Detailed in Visual Studio’s Options window!

We’re getting closer, and MSBuild official documentation has a lot more information on the topic. The key section, for us, will be using MSBuild programmatically.

The goal

Now that we know we can invoke MSBuild ourselves, parsing its output would be complex and still not give us enough data (i.e. no timestamps). Since there’s some kind of API to deal with it programmatically, let’s explore what it can offer!

C# API

At first, when I started building this tool, I was amazed at how little information I could find on how to call MSBuild from C# code. All I found were incomplete or unrelated issues on the MSDN forums, questions without accepted answers at Stack Overflow and a very low number of uses at GitHub.

I spent a lot of time trying to make it work, failing, trying again, mostly working but only for C++ solutions and not C# ones…

In the end, invoking MSBuild just takes these lines:

Dictionary<string, string> globalProperties = new Dictionary<string, string>
{
  { "Configuration", "Debug" },
  { "Platform", "x64" },
};

BuildRequestData data = new BuildRequestData("Z:\\path\\to\\solution.sln",
                                             globalProperties,
                                             null,
                                             new[] { "Rebuild" },
                                             null);

// not providing this default ProjectCollection causes unordered builds
ProjectCollection projectCollection = new ProjectCollection();
BuildParameters parameters = new BuildParameters(projectCollection);
BuildResult result = BuildManager.DefaultBuildManager.Build(parameters, data);
switch(result.OverallResult)
{
  case BuildResultCode.Success:
    // yay!
    break;
  case BuildResultCode.Failure:
    // d'oh!
    break;
}

Simple, right? Well… no.

For it to work you need these BuildRequestData, BuildParameters and BuildManager classes. And this is where the pain started for me.

Reference MSBuild assemblies

By the time I started working on this tool I was using Visual Studio 2015 and I didn’t know where to find any of these assemblies. I only knew they were part of Microsoft.Build packages, as stated in the official docs. So I tried adding a Reference to them via Framework:

Note it says version 4.0.0.0, I didn’t even know which version I needed! This happened like a year ago so I may have forgotten, but I think it worked for a while until I tried to compile a different kind of project. In any case, I ended up referencing assemblies via the Extensions page:

Note it says version 14.0.0.0. Still, no idea why they’re different, but this one worked correctly (and then I knew I was using MSBuild 14).

Bonus: binding redirects

When I thought everything was set, I tried to build my first project and… it failed.

The "Message" task could not be loaded from the assembly Microsoft.Build.Tasks.Core, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a. [...]

This cryptic message hit me like a stone. Why can’t it find it? It compiled and I’m executing the tool now!

Oh, well, turns out you have to apply binding redirects to make it build Visual Studio 2015 solutions. More info on the official docs. I’m a C++ programmer and wasn’t aware of this requirement, I guess .NET programmers are used to it!

I ended up adding this kind of entries to my App.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- [...] -->
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Build" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
        <bindingRedirect oldVersion="0.0.0.0-99.9.9.9" newVersion="14.0.0.0"/>
      </dependentAssembly>
      <!-- more bindings for Microsoft.Build.Framework and other assemblies! -->
    </assemblyBinding>
  </runtime>
</configuration>

Bonus: MSBuild 15 and WPF projects

Everything seemed to work… until I tried to migrate to MSBuild 15 a year later (so I could build Visual Studio 2017 solutions). There’s an official guide to do so (thank you!).

Apparently, versions under MSBuild 15 were bound to their Visual Studio installation and the new recommended way is to pull them via NuGet packages so they’re separate.

With that in place I was able to build VS2017 C++ projects, but had no luck with C# ones.

My GameBoy emulator is built in C++ but uses a Logger built in WPF. Turns out, WPF uses MSBuild itself to process .xaml files (its UI-definition files). That means it tries to spawn and wait for a new MSBuild process within ours! That results in an Unknown build error: object reference not set to an instance of an object. Yes, a null pointer exception is all of the info you have to diagnose it. Good.

The fix is quite simple, however! Add this property to you WPF project:

<AlwaysCompileMarkupFilesInSeparateDomain>false</AlwaysCompileMarkupFilesInSeparateDomain>

And it will use the same MSBuild instance that invoked the build. Voilà!

Get build data

Alright! We’ve managed to make a solution build but we have nothing else to work with! Fear not, the API defines the ILogger interface and the basic Logger class for us! More info on the Build Loggers docs.

Let’s just take everything we can get from the build:

public class AllMessagesLogger : Logger
{
  public override void Initialize(IEventSource eventSource)
  {
    eventSource.AnyEventRaised += OnAnyMessage;
  }

  private void OnAnyMessage(object sender, BuildEventArgs e)
  {
    Console.WriteLine(e.Message);
  }
}

And we can add this logger to the build by modifying the original code with this:

BuildParameters parameters = new BuildParameters()
{
  Loggers = new List<ILogger>() { new AllMessagesLogger() },
};

We’ll now be printing the message associated to every event MSBuild emits. And believe me, there’s a ton of messages!

Still, there are some interesting things we should investigate.

Which events can we hook to?

This is the definition of IEventSource:

public interface IEventSource
{
  event BuildMessageEventHandler MessageRaised;
  event BuildErrorEventHandler ErrorRaised;
  event BuildWarningEventHandler WarningRaised;
  event BuildStartedEventHandler BuildStarted;
  event BuildFinishedEventHandler BuildFinished;
  event ProjectStartedEventHandler ProjectStarted;
  event ProjectFinishedEventHandler ProjectFinished;
  event TargetStartedEventHandler TargetStarted;
  event TargetFinishedEventHandler TargetFinished;
  event TaskStartedEventHandler TaskStarted;
  event TaskFinishedEventHandler TaskFinished;
  event CustomBuildEventHandler CustomEventRaised;
  event BuildStatusEventHandler StatusEventRaised;
  event AnyEventHandler AnyEventRaised;
}

Nice, our old friends the Project, Target and Task! We’ll come back to them in a bit and explore their relationships.

For now, just remember you can bind to different kind of events to get only what you’re interested in.

BuildEventArgs

This is the base class for these events. Of all members it’s got we’re interested in these ones:

Timestamp: the DateTime when the event was emitted (the when).
Message: the text you’ve already read within Visual Studio’s Output panel (the what).
BuildEventContext: the context this event lives in (the where).

Let’s try to understand what’s the BuildEventContext then!

BuildEventContext

This class looks like this:

public class BuildEventContext
{
  public int ProjectContextId { get; }
  public int NodeId { get; }
  public int ProjectInstanceId { get; }
  public int TaskId { get; }
  public int TargetId { get; }

  public long BuildRequestId { get; }
  public int SubmissionId { get; }
  public int EvaluationId { get; }
}

We’re only interested in the first members:

ProjectContextId identifies this context, useful when comparing two of them.
NodeId, after a bit of testing, identifies the logical timeline where it got executed (starts at 1). It has nothing to do with processors, cores or threads so it can get pretty high.
ProjectContextId, TargetId and TaskId identify where this event got executed within a hierarchy. We’ll come back to them shortly.

Alright, I think we can’t postpone exploring Project, Target and Task anymore!

Projects, targets and tasks

MSBuild data is defined in XML files and there’s a hierarchy between definitions. These are the basics:

Project

You may not know it, but that .vcxproj project file you’ve got is an XML file with MSBuild definitions. A Visual Studio 2015 project, for example, looks like:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="14.0" xmlns="[...]">
  <ItemGroup Label="ProjectConfigurations">
    <ProjectConfiguration Include="Debug|x64">
      <Configuration>Debug</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
    <!-- [...] -->
  </ItemGroup>
  <!-- [...] -->
</Project>

The key tag is Project, the root of the document. You can head to the official documentation for more info, but let’s just keep Project in cache.

Target

If we continue exploring this .vcxproj file we’ll find this section:

<!-- [...] -->
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
  <ClCompile>
    <WarningLevel>Level3</WarningLevel>
    <Optimization>Disabled</Optimization>
    <PreprocessorDefinitions>_DEBUG;_LIB;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    <SDLCheck>true</SDLCheck>
    <MinimalRebuild>true</MinimalRebuild>
  </ClCompile>
  <Link>
    <SubSystem>Windows</SubSystem>
    <GenerateDebugInformation>true</GenerateDebugInformation>
  </Link>
</ItemDefinitionGroup>
<!-- [...] -->

There are two tags I want you to pay attention to: ClCompile and Link. Both are specialized Target definitions (we’ll see how in a moment). A Target lives within a Project and groups Task definitions together in a given order. You can read more in the official documentation.

Task

Finally, tasks execute the real logic behind all these definitions. They can create directories (MakeDir task), compile code (CL task) or invoke other projects (MSBuild task), for example.

We could faithfully believe they’re part of a Target, but let’s have a look ourselves, shall we? If you have a basic .vcxproj you won’t see any Task, but you’ll get this entry:

<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />

If you were to find that file, you’d see it has some definitions that point to Microsoft.Cpp.Current.targets and you could continue down the rabbit hole until you get to Microsoft.CppCommon.targets. Inside that file you can find the definition of the ClCompile target:

<!-- [...] -->          
<Target Name="ClCompile"
        Condition="'@(ClCompile)' != ''"
        DependsOnTargets="SelectClCompile">
  <CL Condition="[...]" />
  <CL Condition="[...]" />
  <!-- [...] -->          
</Target>
<!-- [...] -->

And that’s where the CL task gets invoked!

Nice, so a Project can have Target entries and a Target groups Task entries together (one of which can spawn other Project).

This is all great, but what’s a Solution file then?

Solution

If we were to open a Visual Studio 2015 .sln file we’d find some structured text:

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 14
VisualStudioVersion = 14.0.23107.0
MinimumVisualStudioVersion = 10.0.40219.1
[...]
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "GameBoy", "Projects\GameBoy\GameBoy.vcxproj", "{A9A3EC03-A464-4D7C-AC36-857D50CCA937}"
  ProjectSection(ProjectDependencies) = postProject
    {E6C50B23-8563-4980-B0E4-DD4B0570810D} = {E6C50B23-8563-4980-B0E4-DD4B0570810D}
  EndProjectSection
EndProject
[...]
Global
  GlobalSection(SolutionConfigurationPlatforms) = preSolution
    Debug|x64 = Debug|x64
  EndGlobalSection
  [...]
EndGlobal

This defines which projects exist within our solution, their relationship and some other data. If we go back to our Visual Studio’s Developer Command Prompt and type:

set MSBuildEmitSolution=1
msbuild.exe Z:\path\to\solution.sln

We’ll get several .metaproj files we can open. And surprise, they are MSBuild compilant files!

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="14.0"
         xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         InitialTargets="ValidateSolutionConfiguration;ValidateToolsVersions;ValidateProjects"
         DefaultTargets="Build">
  <!-- [...] -->
  <Target Name="GameBoy:Rebuild" ... />
  <!-- [...] -->
</Project>

See the name of the generated Target? This is why we can Rebuild a project (and not the full solution) by executing:

msbuild.exe Z:\path\to\solution.sln /t:ProjectName:Rebuild

Great, so now that we know how MSBuild is structured, we can continue. But before we move on, this is the key takeaway from this section:

Don’t be afraid of diving as deep as you can when facing a new system.

Tying it all together

Let’s take the AllMessagesLogger we created before and run a build. Simplifying a whole lot and gracefully indenting it, we’d get something like:

BuildStarted
  ProjectStarted
    TargetStarted
      TaskStarted
      TaskFinished
      TaskStarted
        ProjectStarted
          [...]
        ProjectFinished
      TaskFinished
    TargetFinished
    TargetStarted
      [...]
    TargetFinished
  ProjectFinished
BuildFinished

Doesn’t this looks like a hierarchy to you?

Build
  Project
    Target
      Task
      Task
        Project
          [...]
    Target

We can now start building our timeline!

Issues

At first, I thought it would be enough to keep stacking events together but soon enough I found not every entry refers to the same hierarchy. You may have two Project building in parallel and events get mixed up.

Here’s where we go back to the BuildEventContext. Remember it had ProjectInstanceId, TargetId and TaskId members? Thanks to that, we can recreate the actual hierarchy!

Each context will populate these values as we get deeper in the hierarchy (i.e. a Project won’t have a TargetId defined), with some exceptions:

The Build is a unique element with no context.
A Project has a ParentEventContext member (defined in the ProjectStartedEvent class). It can be empty (it’s a top-level Project) or reference a Task context.

There are a lot of caveats in the implementation, some inconsistencies (like a Task that spawns a Project but that Task is finished before the Project even starts), and some trial and error.

But, eventually, you get to something!

Google Chrome’s trace viewer

Now that we’ve got our hierarchy in memory we should dump it to some kind of file. Turns out, Google Chrome has this chrome://tracing viewer we can use!

We’ll create a JSON file with a couple of special properties that let us visualize flame graphs. The format and properties are available in this doc.

{
  "traceEvents": [
    { "ph": "B", "pid": 0, "tid": 0, "ts": 0.0, "name": "Hello, flame graph!" },
      { "ph": "B", "pid": 0, "tid": 0, "ts": 2000.0, "name": "I'm a child!" },
      { "ph": "E", "pid": 0, "tid": 0, "ts": 8000.0 },
    { "ph": "E", "pid": 0, "tid": 0, "ts": 10000.0 }
  ]
}

I’ve indented each object to give you a sense of hierarchy, but they’re all children of traceEvents. You can load it into the viewer and this will be the result:

Flame graphs

Let’s take everything, build a couple of projects and see it in action!

Blank project

If you create a blank C++ Win32 Console Application in Visual Studio 2015 with the default configuration, this is how it looks like:

See how it has two CL tasks? One is the pre-compiled header and the other one is the main file. We’ll see why they’re separate in the next post.

GameBoy emulator

This is the trace from my GameBoy emulator (we’ll try to improve it in the next post):

See how it’s using several NodeId and each one has a number of timelines? They represent projects with dependencies!

The top-level Solution executes a MSBuild task to build MainWindows.vcxproj.metaproj (within the same NodeId) and Test.vcxproj.metaproj (in a separate NodeId). Both wait for GameBoy.vcxproj.metaproj to finish (although it’s unreadable in the graph), which in turn waits for another project to build…

It’s a bit complex because of the (seemingly) arbitrary NodeId switches, but that’s how MSBuild schedules project builds!
Remember, these .vcxproj.metaproj are generated from the .sln file!

We can force the execution to build only one project in parallel, and this is the result:

Now, dependencies are much more apparent!

Bruce Dawson’s parallel build

Finally, while investigating slow compile times I read this blog post by @BruceDawson0xB and it helped me a lot (thank you!). He provides the project he used for the post, so I downloaded it and this is its graph:

If you want to know why there are that many CL tasks, I invite you to check his blog post or wait for the next entry in the series!

That’s all for now! We’ve seen how to invoke MSBuild from C# and use its Logger to create our flame graphs.

In the next post we’ll try to understand these graphs, try to diagnose possible issues, and fiddle with some MSVC flags to get extra information (I’m looking at you /Bt+ and /d1reportTime!).

Thanks for reading!

My journey investigating slow compile times in C++

2020-03-25T00:00:00+00:00

You know what’s boring? Processes that take long to complete without you interacting. Making some coffee? I want it already! Doing the laundry? If only it didn’t take hours! Riding a slow elevator? Please, c’mon!

The only good thing about these processes taking long is being able to do something else in the meantime. And you better don’t repeat them very otften!

Programmers repeat one thing way too many times a day: building the project they’re working on. And chances are you’ve come across this awesome webcomic:

Yeah, at first you may smile and think hey, it’s funny. However, when that happens several times a day you start thinking oh, not again instead.

This isn’t a pathology for a single language, platform or business: you may be a game programmer whose project takes several minutes to build (and some more to run!), or a front-end developer whose packaging and deployment process lets you have breakfast while it finishes or you may have some kind of asset in your project you need to process and wish you didn’t need to.

Motivation

I’ve been worrying about slow build times lately. Like, for the last couple of years.

Profiling processes isn’t alien to programmers, sometimes you need to know why something is slow. However, it’s less common to profile build or deployment processes (and we repeat them very often!).

Slow build times cause frustration and discourage both architecture and feature iteration: if I modify this file I get a full rebuild, so I better not touch it. This costs your company increasing amounts of money: people aren’t creating new stuff or improving your systems, and they aren’t happy.

Since I started worrying about these anomalistic build times I’ve tried to learn what kind of things can cause them. I’ve read about ways to investigate them. Found some great tools, used them in our projects, built my own tools. Most of the stuff I’ve studied matches my main development environment: Visual Studio running on Windows, compiling C++ projects with MSVC.

In this series I’d like to share what I’ve learned in the process, including some of those tools or how I’ve used them. Finally, I also want to talk about a way to visualize MSBuild’s build process using Google Chrome’s trace viewer and how it helped me. Have a look at a teaser:

To make this tool, I was inspired by @aras_p from this blog post.
Thank you, Aras!

Disclaimer

Unfortunately, this isn’t a series about how to speed up your compile times, that’s a series on its own. Instead, I want to share what I’ve learned while investigating slow compile times: how MSBuild deals with solutions and projects, how to configure Visual Studio to add extra data or how some MSVC compiler flags can give you a heads up on your development cycle.

You know, the first step when optimizing something is to get some metrics we can compare with, or else we couldn’t know whether it improved. In our case, one question that needs an answer is: how long do the different steps in our build process take?

But first, try to give an estimate to this:

How much time you think you waste waiting for your project to build, each day?

Useful tools to investigate C++ compile times

2020-03-25T00:00:00+00:00

People build the projects they work on several times a day. More often than not, we perform the minimal build cycle: modify some code in a single file, save it and build. Other, sometimes painful, times we have to go through the full build cycle: rebuilding the whole project.

These cycles take some time and what happens under the hood is totally dependent on your development environment and project.

It took me a while to start worrying about these cycles and whether they were long or not. You know why it started? With rising anxiety and a feeling that I wasn’t as productive.

These slow build times come with some costs:

People aren’t happy to stare at progress bar, so they may as well procrastinate and lose focus.
The time they waste waiting is time they aren’t improving your product, either adding new features or iterating.

So, a while ago I set myself on a journey to identify what was causing them. And just so we are on the same page: my main development environment was Visual Studio 2015 running on Windows, building C++ projects with MSVC.

This was (more or less!) the process to answer: how long does it take to build?

Disclaimer: I’ll be illustrating the process with my (unreleased) GameBoy emulator side-project. It’s a small project (we won’t see huge numbers) but it’s not a synthetic test (it mainly consists of C++ but uses C# and C++/CLI for a log window). This post will only cover how to get some build times, we’ll see examples of how to improve them in a later entry in the series.

Visual Studio options

Okay, so we build our project and have just an intuition of how long it takes. But we need some numbers! Just the big picture, nothing too detailed. All I had was Visual Studio’s output panel, and it just said:

1>------ Rebuild All started: Project: LogWindowUI, Configuration: Debug x64 ------
1>  [...]
2>------ Rebuild All started: Project: LogWindowBridge, Configuration: Debug x64 ------
2>  [...]
3>------ Rebuild All started: Project: LogLibrary, Configuration: Debug x64 ------
3>  [...]
4>------ Rebuild All started: Project: GameBoy, Configuration: Debug x64 ------
4>  [...]
5>------ Rebuild All started: Project: MainWindows, Configuration: Debug x64 ------
5>  [...]
6>------ Rebuild All started: Project: Test, Configuration: Debug x64 ------
6>  [...]
========== Rebuild All: 6 succeeded, 0 failed, 0 skipped ==========

I could take a stopwatch and start it as soon as I hit the build button, but not much more.

After checking Visual Studio’s documentation and Stack Overflow questions on this matter, I ended up on this Microsoft Docs site.

Let’s explore some options.

Build and Run

This page has several useful options, and we’ll come back to it later on, but let’s just take a look at MSBuild project build output verbosity.

If we increase it from Minimal to Normal, we get this output:

1>------ Rebuild All started: Project: LogWindowUI, Configuration: Debug x64 ------
1>Build started DD/MM/YYYY HH:MM:SS.
1>  [...]
1>Build succeeded.
1>
1>Time Elapsed 00:00:02.47
[...]
6>------ Rebuild All started: Project: Test, Configuration: Debug x64 ------
6>Build started DD/MM/YYYY HH:MM:SS.
6>  [...]
6>ClCompile:
6>  Main.cpp
6>  [...]
6>Link:
6>  [...]
6>Build succeeded.
6>
6>Time Elapsed 00:01:36.92
========== Rebuild All: 6 succeeded, 0 failed, 0 skipped ==========

Great! We now have some data to work with! There are three interesting additions:

Build started DD/MM/YYYY HH:MM:SS: the real output doesn’t show those letters, but actual numbers. We now know when this project started building.
Time Elapsed HH:MM:SS.ms: we now know how long it took to build this project.
ClCompile, Link and other new sections: they are what MSBuild calls targets. We’ll come back to them in depth later on.

If we increase the previous option from Normal to Detailed the output will be more verbose, it will include more info on those targets and also add tasks to the log. All this data will prove extremely useful in the future, but they don’t add extra information on how long it takes to build. Even worse: the build itself gets slower as you increase this log level.

VC++ Project Settings

This page also has a handful of useful options, but the most important one for us is Build Timing.

When we enable it and build our project, we get something like this:

[...]
6>------ Rebuild All started: Project: Test, Configuration: Debug x64 ------
6>Build started 25/03/2020 11:03:11.
6>  [...]
6>Project Performance Summary:
6>    96955 ms  F:\Development\GameBoyEmulator\Projects\Test\Test.vcxproj   1 calls
6>              96955 ms  Rebuild                                    1 calls
6>
6>Target Performance Summary:
6>        0 ms  AfterRebuild                               1 calls
6>       [...]
6>       41 ms  WarnCompileDuplicatedFilename              1 calls
6>      182 ms  CoreCppClean                               1 calls
6>      489 ms  SetCABuildNativeEnvironmentVariables       1 calls
6>     2332 ms  Link                                       1 calls
6>    93807 ms  ClCompile                                  1 calls
6>
6>Build succeeded.
6>
6>Time Elapsed 00:01:36.92
========== Rebuild All: 6 succeeded, 0 failed, 0 skipped ==========

Now we’re talking! This output includes the time spent on each target as reported by MSBuild (we still don’t know what’s a target!), so we know where it’s getting slow. Are C++ file compilations slow? Or maybe link times are too high?

By the way, this option and the one from the previous point aren’t related to each other: you can leave the former as Minimal and enable the latter.

So far, so good! What should we do with this, then? Should we try to parse this output and build a timeline already? Let’s continue looking for tools first.

Visual Studio extensions

A lot of the times you want some tool, chances are someone already built it. And, in this Visual Studio ecosystem we’re bound to, we should take a look at Extensions.

Let me discuss a couple of the ones I found more suitable for my needs.

Build Monitor

Build Monitor is an interesting extension that supports, at the time of writing, up to Visual Studio 2015.

Turns out, Visual Studio Extensions’ SDK lets you get notified of some MSBuild key events: when did the build start? when did a project finish? And that’s what this extension does. Build your solution and you’ll get this kind of output:

Solution loaded:    GameBoyEmulator
------------------------------------------------------------
 - 00h 00m 05s 298ms  -- LogWindowUI --
 - 00h 00m 03s 594ms  -- LogWindowBridge --
 - 00h 00m 01s 555ms  -- LogLibrary --
 - 00h 00m 06s 393ms  -- GameBoy --
 - 00h 00m 04s 852ms  -- MainWindows --
 - 00h 01m 38s 037ms  -- Test --
[1] Time Elapsed: 00h 01m 55s 300ms     Session build time: 00h 01m 55s 300ms

While Visual Studio is open, it will keep track of the session, adding up each compilation you do. What if you close Visual Studio? The session is gone, but the coolest thing is that sessions are persisted to a JSON file! This way, you can always come back to them.

This is good enough to compare two builds together!

Tool idea

Have all programmers (or everyone using Visual Studio!) in your team install this extension. Build a script that collects each JSON file at the end of the day, sends it to a server and clears the local one. Later on, aggregate all files together, tag data as necessary and you have the time your team is wasting by building the project (including how often each project is built)! If you plot it, you can even see some trends over time or spikes when a big code change happens.

This is the real time people waste, not how long your build server takes to make a build.

Be careful, tho! The results may be daunting at first (i.e. too much time wasted)!

Parallel Builds Monitor

If you liked the previous one but prefer a more graphical approach, then take a look at Paralel Builds Monitor. It’s got better version support, up to Visual Studio 2019 (at the time of writing).

Although this extension can also export build data to a CSV file, it’s the Gantt chart that I like the most. Not only it tells you which projects are building in parallel (you may think you removed some dependency but didn’t!), it lets you quickly see how long did each project take to build. Absolutely recommended!

This graph can be docked like another panel, so be careful if you’re an anxious person: seeing how much time is gone on each build may be unwise for you!

MSVC compiler flags

Nice, so we’ve got some big numbers and we know which projects take longer to build. But we still don’t know where time is spent! Fortunately, MSVC has some compiler options that can help us investigate further:

/Bt+

This flag applies to the compiler and produces this sample output when turned on:

1>  File.cpp
1>  time(Z:\path\to\compiler\c1xx.dll)=0.39427s < 1617765075910 - 1617766036302 > BB [Z:\path\to\File.cpp]
1>  time(Z:\path\to\compiler\c2.dll)=0.06475s < 1617766053824 - 1617766211553 > BB [Z:\path\to\File.cpp]

This shows the time spent on the front-end (c1xx.dll as we’re compiling C++ code, it would be c1.dll when compiling C code) and on the back-end (c2.dll). It’s extremely useful to narrow the long times down.

You can find more info on the official C++ devblogs.

/time+

This one applies to the linker and gives this kind of output in a typical Debug build:

5>  Linker: Pass 1: Interval #1, time = 0.78976s [Z:\path\to\program.exe]
5>  Linker:   Wait PDB close Total time = 0.32035s PB: 7331840 [Z:\path\to\program.exe]
5>  Linker: Pass 2: Interval #2, time = 0.48362s [Z:\path\to\program.exe]
5>  Linker: Final Total time = 1.27344s < 1618429762951 - 1618432864884 > PB: 7331840 [Z:\path\to\program.exe]

While you’ll get this info in a typical Release build:

5>  Linker: (Z:\path\to\compiler\c2.dll)   LTCG Load Main.obj: Total time = 0.04933s PB: 35180544 [Z:\path\to\program.exe]
5>  [...]
5>  Linker: (Z:\path\to\compiler\c2.dll)   OptRef Total time = 0.00091s PB: 40161280 [Z:\path\to\program.exe]
5>  Generating code
5>  [...]
5>  Finished generating code
5>  Linker: (Z:\path\to\compiler\c2.dll)   LTCG CodeGen Total time = 2.58961s PB: 57692160 [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll)   OptRef Total time = 0.00077s PB: 60547072 [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll)   OptIcf Total time = 0.00857s PB: 60551168 [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll) Pass 1: Interval #1, time = 3.56095s [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll)   Wait PDB close Total time = 0.14563s PB: 59539456 [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll) Pass 2: Interval #2, time = 0.23024s [Z:\path\to\program.exe]
5>  Linker: (Z:\path\to\compiler\c2.dll) Final Total time = 3.79130s < 1618638531573 - 1618647766698 > PB: 59539456 [Z:\path\to\program.exe]

Remember: by default Release configurations have Link Time Code Generation turned on to optimize further!

You can also find more info here.

/verbose

Although this flag doesn’t give you elapsed times, it’s also very useful if you want to investigate why link times are slow. I haven’t had much luck with it, but it’s interesting to see the amount of data the linker processes.

/d2cgsummary

This flag enables a cool behavior on the compiler’s back-end (hence the d2) from Visual Studio 2015 onwards. I learned about this flag thanks to Aras, and I invite you to check his blog post for the full explanation!

The key takeaway is that you get a summary of the code generation which includes anomalistic compile times and points you to the functions you should investigate!

Although the slow build times I have investigated weren’t related to code generation, this flag is extremely useful!

/d1reportTime

Again, I learned about this flag thanks to Aras, so please check his post for all of the info. This one applies to the compiler’s front-end (hence the d1).

The bad thing about this flag is the huge amount of data it gives… and the cool thing is the huge amount of data it gives! You get, separately, the time spent when including files (hierarchically!), the one declaring classes and the same for functions. It also has a summary on each section with the top sorted culprits so you can find useful data more easily.

If only we could visualize its output in a more graphical way…

This is a teaser of the next post in the series!

It’s a pity this flag is only available on Visual Studio 2017 onwards, because at the time I started investigating build times we were bound to VS2015.

External tools

Now that we know how to get high level and low level build times, we should check some other tools to complement this data or help us diagnose why they’re happening.

So far we’ve been talking about C++ development, and that means talking about #include clauses. We’ve also mentioned MSVC compiler execution consists of two phases:

Front-end: reads code files, parses them, executes preprocessor directives (i.e. #include clauses) and applies semantic analysis.
Back-end: takes front-end’s output and performs code generation.

When you have large files and a lot of dependencies via #include, you’re bound to have long front-end execution times. When that’s the case, it’s useful to see those connections.

doxygen

Yes, the one that helps you document your code. doxygen comes with a handful of tools, and one of them is building dot graphs.

This example is extracted from doxygen’s docs, but trust me when I tell you it can handle huge dependency graphs:

The cool thing about these graphs is they’re interactive when you use the html version (check previous link) and you can navigate at will. There’s even an option to configure the maximum inclusion depth in a single graph, so it has you covered!

Header Hero

A long while ago I found this blog post (which also gives very useful techniques to reduce your file dependencies!) and had a try. But it wasn’t until I read @aras_p in this other blog post that I put it to good use.

This tool has proven very useful, although it can take some trial and error to get the data you’re looking for when your project is very complex. I highly recommend you check Aras’ additions to the tool as well!

When you get it working, take one of the biggest contributors or hubs, click on it and start exploring which files include it. Should they? Can you refactor your code so they don’t (i.e. forward declarations, remove nested classes or enums)?

Visual Studio Code Map

Unfortunately this tool is only available in a Visual Studio Enterprise installation, so I haven’t tested it myself (although it looks promising!).

From what I see in the official documentation it produces complex graphs with several layers of data, but I guess some of that complexity may be configurable.

The great thing about finding this tool was learning there’s a graph format you can use for your own tools, and Visual Studio will be the viewer! You may use it for #include clauses, asset dependencies or whatever fits in a directed graph!

Other tools with dependency graphs

I also found some other Visual Studio extensions that give you inclusion dependency graphs like Include Toolbox (VS2017+) or Visual Assist (has a trial version).

I’m sure there are a lot more!

MSBuild Structured Log Viewer

More recently, Kirill Osenkov pointed me to MSBuild Structured Log Viewer.

MSBuild 15.3 onwards lets you dump the full log it produces into a binary file you can read with that viewer. This includes project-target-task hierarchies, clicking on a target to see its definition in code and a great search tool. You can even replay a build by feeding that same binary file to MSBuild in case you want to do something with it!

I encourage you to give it a try! This tool shares one of the principles with the one we’ll talk about in the next post in the series, so it feels very familiar to me!

C++ Build Insights

This one is even more recent, available from Visual Studio 2019 according to Kevin’s blog post on the official C++ developers blog.

To put it simply, the new versions of MSVC compiler and linker emit ETW events and they’ve added specially tailored panels to the WPA so you can explore them. This means you’ll get exact timestamps, which is awesome when you want to investigate!

And even more, Kevin shared this post not even a month ago with the release of an SDK to help you build your own tools on top of that info! This looks like the tool I was looking for, so thank you for the release!

At the time of writing, you need to register to the Windows Insider Program to get the latest version, but it’s a quick process. Have a try and build great tools with it!

After a couple of years investigating these build times it’s very possible I forgot about some tool or compiler flag, but this was more or less my process.

We’ve mentioned MSBuild several times. We talked about targets and tasks. Further investigation led me to building my own tool so we can get a flame graph out of MSBuild! Visualizing data in a graphical may be key! That will be the next post in the series.

Thank you for reading, hope some of these tools help you!

Log window from scratch: C++ to C# interoperability

2018-06-27T00:00:00+00:00

Hello and welcome to the last entry in the series!

So far we’ve gone through the process of building a WPF live log window we could use from other C# projects. We made it a Class Library and used it from a host program also written in C#.

This time, we’ll learn how we can have a C++ project use it. Yeah, that’s right: we’ll be calling C# from C++!

Let’s do it!

Overview

Before we dive into creating projects, configurations and code, let’s have a look into what we’ll need.

First of all, our WPF log window is C# code. If we were to categorize it by how memory is dealt with, we’d call it managed. This is because we create memory via new and the Garbage Collector will know when to free it.

Second, our new host program will be C++ code. However, because there’s no Garbage Collector of any sort, we can call it unmanaged. We can also call it native code.

So, you could say: how can we invoke C# code from C++ code? The short answer is you can’t, directly. Fortunately for us, there’s a long answer!

C++/CLI

There’s a language called C++/CLI that we could basically say is C++ with new functionality to deal with .NET languages via Common Language Infrastructure. We could call this kind of code mixed.

Knowing this, we could have C++/CLI code sitting in between of C++ and C#, like in this diagram:

With this picture in mind, these will be the steps:

Create a C++ project with the typical Hello, world!
Create a C++/CLI project with a not-so-typical Hello, mixed world!
Connect these two so we can invoke the mixed one from the unmanaged one.
Connect the mixed project with our managed WPF one.

C++ host program

Let’s start by adding a new C++ project to our solution: right-click it, then Add -> New project… and choose Win32 Console Application from the Visual C++ filter. Give it a descriptive name like NativeHostProgram. From the wizard select Empty project.

Now, create the typical Main.cpp:

int main(int argc, char **argv)
{
    std::cout << "Hello, world!" << std::endl;
    return 0;
}

Set this project as the startup project and run it. So far, so good. Next step?

C++/CLI bridge project

We’ve set our C++ project as the main one, so we can’t do the same for this C++/CLI project. Instead, we’ll use it from the native one, and that means it must be a library.

Again, right-click on the solution then Add -> New project… but this time, under Visual C++ select the CLR group. The only project type we’re given that matches our requirement is a Class Library, so let’s create that one. Give it a nice descriptive name, like LogWindowMixedBridge.

By doing so, Visual Studio will also create LogWindowMixedBridge.h and LogWindowMixedBridge.cpp for us. I prefer deleting them to start from scratch once again.

Let’s instead create a nice and simple Bridge.h with this code:

#pragma once

#include <iostream>

void helloMixedWorld()
{
    std::cout << "Hello, mixed world!" << std::endl;
}

To be able to invoke this helloMixedWorld we must first connect the two projects together. But, how can we do that?

Referencing bridge from native

When we chose Class Library while creating our C++/CLI project we were in fact creating a DLL, so we must follow the usual process for linking DLLs. This time, we’ll use the Implicit Linking method.

Let’s change our Bridge.h file to:

#pragma once

#ifdef MIXED_MODE_DLL_EXPORT
#define MIXED_BRIDGE_API __declspec(dllexport)
#else
#define MIXED_BRIDGE_API __declspec(dllimport)
#endif

MIXED_BRIDGE_API void helloMixedWorld();

And a Bridge.cpp file with:

#include "Bridge.h"

#include <iostream>

MIXED_BRIDGE_API void helloMixedWorld()
{
    std::cout << "Hello, mixed world!" << std::endl;
}

Which will yield this error:

Error: a function declared ‘dllimport’ may not be defined.

This is because MIXED_MODE_DLL_EXPORT isn’t defined, which means we’re declaring the helloMixedWorld as dllimport. However, DLL projects must use dllexport for their functions so other projects use dllimport. So, where can we define MIXED_MODE_DLL_EXPORT?

Right-click the LogWindowMixedBridge project and select Properties. Under C/C++ > Preprocessor > Preprocessor definitions then add MIXED_MODE_DLL_EXPORT.

With this, the error has gone away!

Now, compile the LogWindowMixedBridge project alone. It should complete with no errors and will have created two files we’re interested in: LogWindowMixedBridge.dll and LogWindowMixedBridge.lib. These, alongside the Bridge.h file are the three things we need to implicitly link this DLL.

Implicit Linking

We said we needed three things to be able to reference this DLL from our NativeHostProgram:

Have access to the .h files with declarations.
Have access to the .lib file to perform the link.
Have access to the .dll file.

With this, it’s pretty much working as a static library although it is a DLL.

Right-click NativeHostProgram, select Properties and perform these steps (we’re using paths from the default Visual Studio configuration):

Update C/C++ > Additional Include Directories: %(SolutionDir)LogWindowMixedBridge\;%(AdditionalIncludeDirectories).
Update Linker > Additional Dependencies > Input: LogWindowMixedBridge.lib; and all of the previous values.
Update Linker > General > Additional Library Directories: $(SolutionDir)\$(Configuration)\;%(AdditionalLibraryDirectories) for Win32 and $(SolutionDir)$(Platform)\$(Configuration)\;%(AdditionalLibraryDirectories) for x64.

Last, but not least, we must tell the solution that NativeHostProgram depends on LogWindowMixedBridge. To do so, right-click the solution, select Properties and under Project Dependencies select NativeHostProgram and check LogWindowMixedBridge. From now on, when we compile the solution it will first compile LogWindowMixedBridge and then NativeHostProgram.

Well, we’ve linked everything so we can go back to Main.cpp in NativeHostProgram and update it with:

#include <Bridge.h>

int main(int argc, char **argv)
{
    helloMixedWorld();
    return 0;
}

Run the solution and you’ll have an impressive message in your console saying the Hello, mixed world! message!

Phew! It was intense! Well done :)

Connecting C++/CLI to C#

Nice! We know how to connect C++ with C++/CLI, but how about going from C++/CLI to C#? The answer is in fact pretty simple: reference it!

To do so, expand our LogWindowMixedBridge project, right-click References, select Add Reference… and then check our LogWindowUI project (which, if you remember from previous entries, is our C# WPF project).

Now we can update our Bridge.cpp file with this code:

#include "Bridge.h"

MIXED_BRIDGE_API void helloMixedWorld()
{
  LogWindowUI::LoggerUI::Initialize(0, 0, 500, 300);
  LogWindowUI::LoggerUI::Instance->Add(0.0f, "DEBUG", "TEST", "Hello, WPF!");

  // required so we can see the window for a bit
  System::Threading::Thread::Sleep(5000);
}

Run it and you’ll get this:

Impressive! It may look like no big deal, but we’ve shown our log window from a code that started at C++! Now, for the last bit!

Bridge revisited

Alright, we’re awesome because we communicated C++ with C#, but for now it’s just a function! Why don’t we design it a bit and make our C++ program configure the C# WPF window and log some stuff?

Our LoggerUI class, which belongs to C#, has methods to deal with the window like Initialize, ConfigureSystems or Add. Wouldn’t it be awesome to have some kind of wrapper in our C++/CLI project to call them from C++? We’re doing that now.

Let’s modify our Bridge.h file with this:

#pragma once

#ifdef MIXED_MODE_DLL_EXPORT
#define MIXED_BRIDGE_API __declspec(dllexport)
#else
#define MIXED_BRIDGE_API __declspec(dllimport)
#endif

#include <vector>

class MIXED_BRIDGE_API Bridge
{
public:
  Bridge(int x, int y, int w, int h);
  ~Bridge();

  void configureSystems(std::vector<const char *> systems);
  void configureLevels(std::vector<std::pair<const char *, const char *>> levels);

  void log(const char *level, float timestamp, const char *system, const char *message) const;
};

Notice how we can export a whole class with the DLL, not just free functions. As you can see, its methods mostly map those at LoggerUI. Why don’t we see some of the implementations?

Bridge::Bridge(int x, int y, int w, int h)
{
  LogWindowUI::LoggerUI::Initialize(x, y, w, h);
}

Bridge::~Bridge()
{
  LogWindowUI::LoggerUI::Destroy();
}

Nothing pretty fancy here, right? What about configureSystems, for example?

void Bridge::configureSystems(std::vector<const char *> systems)
{
  typedef System::Collections::Generic::List<System::String ^> tManagedStringList;
  tManagedStringList ^systemsManaged = gcnew tManagedStringList();
  for (const char *s : systems)
  {
    systemsManaged->Add(gcnew System::String(s));
  }

  LogWindowUI::LoggerUI::Instance->ConfigureSytems(systemsManaged);
}

This is what our Bridge is all about: translating stuff from C++ to C#. See how we’re converting the const char * to String? Okay, but you may say: what’s that String ^ or gcnew?

The ^ symbol represents a pointer to managed memory, and that memory must be created somewhere. That’s where gcnew comes into play: creates memory handled by the Garbage Collector. We could say they are the managed counterparts of * and new.

LoggerUI as a Singleton

By now you’ve already noticed we’re enforcing having a single LoggerUI instance because it’s a Singleton. However, nothing prevents us from creating several Bridge instances! We could create two of them and then an assert would trigger because we’d be trying to call LoggerUI::Initialize twice!

We could fix it by ditching out the Singleton pattern at this level and having our Bridge wrapper have a private LoggerUI member. However, it’s a bit more convoluted than I wanted to dive into when I started this post so we’ll keep it out of scope.

Long story short, it requires creating a BridgePrivate class to be used by Bridge (pretty much like the PIMPL) with a member of type gcroot<LoggerUI ^>. The reason is we can’t expose a C++/CLI class to C++ with pointers to managed memory, so we must hide it in the private class.

Putting it all together

The only thing we’re missing is calling all this from C++! Let’s update Main.cpp with this:

#include "Bridge.h"

#include <chrono>
#include <thread>

int main(int argc, char **argv)
{
  std::vector<const char *> systems;
  systems.push_back("TEST");

  std::vector<std::pair<const char *, const char *>> levels;
  levels.push_back(std::pair<const char *, const char *>("DEBUG", "#000000"));

  Bridge *bridge = new Bridge(0, 0, 400, 200);
  bridge->configureSystems(systems);
  bridge->configureLevels(levels);

  bridge->log("DEBUG", 0.0f, "TEST", "Hello from native C++!");

  std::this_thread::sleep_for(std::chrono::seconds(20));
  delete bridge;

  return 0;
}

And this is the result:

What if we configured more levels and more systems? What if we logged messages with random level and system? What if we recorded it into a GIF?

Congratulations! You’re now logging messages from C++ to a WPF window written in C#!

Wow, so that’s the end of the series! We’ve gone through the process of creating a WPF live log window which we can use from C# and C++ projects. We’ve started from nothing and got to the previous GIF showing the result, and what a result! :)

The only thing that’s left is using it in other real projects! What are you waiting for?

Thanks a lot for reading!

Log window from scratch: handy functionality and configuration (part 2)

2018-06-19T00:00:00+00:00

In the previous post we went through some interesting functionalities for our log window but we were missing the, arguably, most important one.

Should we start working on it? Oh yes, and you’ll love it.

Log levels

This wouldn’t be a proper log window if we didn’t have log levels (debug, warning, error) and colors for them!

Again, we’ll let the host program decide which log levels it uses, their names and even the colors. And yes, you’ve guessed right, we’ll do that in the LoggerUI singleton.

But before we start, what’s the relationship between different log levels? How about using the common one: each level represents an integer (higher means more severity). Like in this example:

Imagine we had three log levels: debug, warning and error. We could use debug to log data or flow, warning would let us know of non-blocking incompatibilities or to diagnose future issues and error would flag stuff that must be fixed although we managed to continue executing the program. If we set our current log level to warning, we would also see error ones because their severity is higher.

Oh, and for it to work we must remember to update our LogEntry messages to have a log level as well!

Configuration from the host

We’ve decided each log level is modelled as a name and a color. The severity can be implicit based on the configuration: the first ones are less important. What about adding this to LoggerUI:

public void ConfigureLevels(List<Tuple<string, string>> levels)
{
    Debug.Assert(m_application != null);

    m_application.Dispatcher.BeginInvoke((Action)delegate
    {
        Debug.Assert(m_application.MainWindow != null);

        (m_application.MainWindow as MainWindow).ConfigureLevels(levels);
    });
}

We could’ve created a new class or struct as the Model for each log level but we’ve used Tuple<string, string> instead to get to the point. The first string is the name, the second one is the hexadecimal representation of the color.

We could call it like so:

LoggerUI.Instance.ConfigureLevels(new List<Tuple<string, string>>
{
    Tuple.Create("DEBUG",   "#000000"),
    Tuple.Create("WARNING", "#B8860B"),
    Tuple.Create("ERROR",   "#FF0000")
});

Now, we’d have to create MainWindow.ConfigureLevels, but let’s create a new ViewModel before.

ViewModel

Similarly to the LogSystem one we created in the previous post, let’s define LogLevel as:

public class LogLevel
{
    public string Name  { get; set; }
    public Brush Color  { get; set; }
    public int Severity { get; set; }
}

Again, let’s have a new LogLevels list in our MainWindow to contain all of the levels. And we could populate it from this method:

public void ConfigureLevels(List<Tuple<string, string>> levels)
{
    for(int i = 0; i < levels.Count; ++i)
    {
        LogLevels.Add(new LogLevel
        {
            Name = levels[i].Item1,
            Color = (Brush) new BrushConverter().ConvertFromString(levels[i].Item2),
            Severity = i
        });
    }
}

You’ll understand why we’ve used Brush instead of Color in a moment.

Cool! Now we’ve got all of the values in a list. What’s next?

Add log level to LogEntry

Right, until now all of our LogEntry messages only had a timestamp, a system and the message itself. Now, we need to add the name of the log level it was logged with. I’ll let you go through this one, just add a new string property and we’re ready.

Layout update

We’ve established the relationship between log levels. Following our example, debug < warning < error. So, we only want to select the minimum level we’re showing and the ones over it should show as well. This looks like a radio button to me.

The layout of our window would look like this with this addition:

So, once again, let’s skip some XML attributes and update the layout:

<Window ...>
    <DockPanel>
        <GroupBox ... />
        <GroupBox ...>
            <StackPanel>
                <ListView Name="Systems" ... />
                <ListView Name="Levels"
                          ScrollViewer.HorizontalScrollBarVisibility="Disabled"
                          BorderThickness="0">
                    <ListView.ItemsPanel>
                        <ItemsPanelTemplate>
                            <WrapPanel Orientation="Horizontal"></WrapPanel>
                        </ItemsPanelTemplate>
                    </ListView.ItemsPanel>
                    <ListView.ItemTemplate>
                        <DataTemplate>
                            <RadioButton Content="{Binding Name}"
                                         GroupName="LogLevels"
                                         Foreground="{Binding Color}">
                            </RadioButton>
                        </DataTemplate>
                    </ListView.ItemTemplate>
                </ListView>
            </StackPanel>
        </GroupBox>
        <ListView x:Name="LogEntryList" ... />
    </DockPanel>
</Window>

The Foreground property expects a Brush instead of a raw Color, so that’s the reason why we used one.

Tweaking the sample messages a bit and taking a screenshots looks like this:

As you can see, none of the elements is checked. We’ll work on that soon. But first, let’s add some colors to the messages!

Message coloring

Now that all of the messages have a log level themselves, we could style the LogEntryList so each row has a color matching the configuration. That way, we can skip adding a new column with the name of the log level.

If all of our log levels and their colors were static resources we’d be defining them in XAML. Since we’re configuring it externally, we must create the styles programmatically.

To do that, let’s update our MainWindow.ConfigureLevels like so:

public void ConfigureLevels(List<Tuple<string, string>> levels)
{
    // create log levels
    for(int i = 0; i < levels.Count; ++i)
    {
        LogLevels.Add(new LogLevel
        {
            Severity = i,
            Name = levels[i].Item1,
            Color = (Brush) new BrushConverter().ConvertFromString(levels[i].Item2)
        });
    }

    // style ListView based on the data from the log levels
    Style logListStyle = new Style();
    logListStyle.TargetType = typeof(ListViewItem);
    foreach(LogLevel level in LogLevels)
    {
        DataTrigger trigger = new DataTrigger();
        trigger.Binding = new Binding("Level");
        trigger.Value = level.Name;
        trigger.Setters.Add(new Setter(ListViewItem.ForegroundProperty, level.Color));

        logListStyle.Triggers.Add(trigger);
    }

    LogEntryList.ItemContainerStyle = logListStyle;
}

A Style, as we can see, has the concept of a DataTrigger. It takes a Binding and a Value to test against, and if it matches then the Setters are applied. In our case just a text color, but we could use other styling. Finally, we assign the whole style to the ItemContainerStyle, which is the one applied to the ListViewItem entries.

This is the result:

Looking nice, huh? However, there’s something missing, isn’t it?

Filtering

Alright, alright. We’re happy with the results but we still can’t do anything with these radio buttons! Let’s see what’s left.

We said we’d be adopting the usual each log level has a severity and we can show all of the logs with a higher severity level than the selected one. We’ve got as many radio buttons as log levels, and each one represents a severity (although we aren’t displaying the number itself).

So, it looks like we only want one variable: the currently selected log level. But, how do we go from many radio buttons to a single value?

First of all, let’s create a new variable in MainWindow:

public int CurrentLogLevelSeverity { get; set; }

Now, let’s update our LogLevel class to this:

public class LogLevel : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    public string Name     { get; set; }
    public Brush  Color    { get; set; }
    public int    Severity { get; set; }

    private bool m_selected;
    public  bool Selected
    {
        get
        {
            return m_selected;
        }
        set
        {
            m_selected = value;
            if (PropertyChanged != null)
            {
                PropertyChanged(this, new PropertyChangedEventArgs("Selected"));
            }
        }
    }
}

It looks pretty similar to LogSystem, doesn’t it?

Now, MainWindow.ConfigureLevels:

public void ConfigureLevels(List<Tuple<string, string>> levels)
{
    // create log levels
    for(int i = 0; i < levels.Count; ++i)
    {
        LogLevel entry = new LogLevel
        {
            Severity = i,
            Name = levels[i].Item1,
            Color = (Brush)new BrushConverter().ConvertFromString(levels[i].Item2)
        };

        entry.PropertyChanged += OnLevelSelectedChanged;

        LogLevels.Add(entry);
    }

    // [...]
}

So, whenever the LogLevel changes (when the user interacts with the radio buttons) we’ll get notified. And what do we do then?

private void OnLevelSelectedChanged(object sender, PropertyChangedEventArgs args)
{
    LogLevel level = sender as LogLevel;

    if(level.Selected)
    {
        CurrentLogLevelSeverity = level.Severity;

        FilteredLogEntries.Refresh();
    }
}

Because the Selected property will be modified for both the radio button that’s getting selected and the one that’s being deselected, we’re just interested in the former. We update the log level and ask the filter to be re-evaluated.

But, we must add the log level to the filter or it won’t do anything different!

private bool LogEntriesFilterPredicate(object item)
{
    LogEntry entry = item as LogEntry;

    // filter out systems
    if(LogSystems.Any(s => s.Name == entry.System && !s.Enabled))
    {
        return false;
    }

    // filter out levels
    LogLevel level = LogLevels.First(l => l.Name == entry.Level);
    if(level != null && level.Severity < CurrentLogLevelSeverity)
    {
        return false;
    }

    return true;
}

Each message has the name of the log level it was logged with, and we have the severity we want to test against. So, we find out which LogLevel matches the given name and only keep those messages with a matching or greater severity. Bonus points if you can optimize this to prevent looking for the LogLevel.

And that’s it! Let’s take some screenshots!

Not bad! But can we improve the layout so it looks better?

What about adding more systems and more log levels from the host program? Don’t cheat, you can’t modify the WPF project!

Congratulations, reader! You’ve got a configurable log window of your own! :)

Oh well, I guess that’s it! With this post and the previous one we’ve gone through some useful features: positioning, auto-scrolling, configuring log systems and log levels, filtering… Which other features you’d like to add? I encourage you to do it!

I hope this series helps you create your own version of the log window and use it in the future on your side projects!

In the next post we’ll see how we can have a C++ host program that uses this log window.

Thanks for reading!

Log window from scratch: handy functionality and configuration (part 1)

2018-06-17T00:00:00+00:00

Welcome to a new entry in the Log window from scratch series, dear reader!

In the previous entry we converted our WPF window into a Class Library and created a host program that used it to display some sample messages. This time, we’ll expose some configuration to the host program and add some new functionality to the window. This way we can support several projects and make it more useful.

Ready? Set? Go!

Window position and dimensions

Let’s start with a simple one: setting the position and size of our window.

Imagine you are developing a game and it has a window where it gets rendered. You don’t want your log window to be created over it (or under it!), but side to side instead.

We could add two new methods to our LoggerUI called SetPosition and SetSize, or we could pass these parameters to the Initialize method and have them set from the start. For now, let’s pass them to Initialize since we don’t plan on resizing it programatically after it’s shown.

First, update Initialize like so:

public static void Initialize(int x, int y, int w, int h)
{
    Debug.Assert(m_instance == null, "LoggerUI already initialized");
    m_instance = new LoggerUI(new Rect(x, y, w, h));
}

So we can call it like so:

LoggerUI.Initialize(0, 0, 800, 200);

Of course, we’d need to update our LoggerUI’s constructor to match the new parameter:

private LoggerUI(Rect dimensions)
{
    // application and window need their own thread, so we create it
    AutoResetEvent windowCreatedEvent = new AutoResetEvent(false);
    Thread t = new Thread(() =>
    {
        m_application = new App();

        MainWindow window = new MainWindow();

        // set window dimensions
        window.WindowStartupLocation = WindowStartupLocation.Manual;
        window.Left = dimensions.Left;
        window.Top = dimensions.Top;
        window.Width = dimensions.Width;
        window.Height = dimensions.Height;

        m_application.MainWindow = window;
        m_application.MainWindow.Show();

        // notify they are created before we block this thread
        windowCreatedEvent.Set();

        m_application.Run();
    });
    t.SetApartmentState(ApartmentState.STA);
    t.Start();

    // wait until the application and window are created
    windowCreatedEvent.WaitOne();
}

The only addition has been the block where we set the dimensions.
And this is it! We can now position and size it however we want.

Bonus idea: we could ask the window about these properties (which would be updated when resizing or moving the window) from the host program and persist its values so we could start from the last configuration when we launch the program again.

Auto-scroll to bottom

This will be the first addition that will modify our window’s layout.

If you were to use the window as it is now, you would notice the scroll isn’t moving unless you do it manually. This may be useful if you are reading some of the messages, but most of the time you may want it to scroll to the last message automatically.

We’ll be adding a checkbox to our window that lets us activate or deactivate this functionality. This isn’t something the host program can choose to have or not, but something that’s built-in. So our layout will be something like this:

Layout update

Let’s start with the built-in block. This is the updated MainWindow.xaml file with some parts commented out for the sake of brevity:

<Window ...>
    <DockPanel>
        <GroupBox DockPanel.Dock="Top"
                  x:Name="BuiltInConfigurationGroup"
                  VerticalAlignment="Top"
                  BorderThickness="0">
            <CheckBox x:Name="AutoScrollCheckBox"
                      Content="Auto-scroll"
                      HorizontalAlignment="Right"
                      VerticalAlignment="Center"/>
        </GroupBox>
        <ListView DockPanel.Dock="Top" ...>
            <!-- other properties -->
        </ListView>
    </DockPanel>
</Window>

As you can see, we’ve wrapped everything into a DockPanel which allows us to resize the window and keep every control stretched as we want while keeping some parts fixed. This is how it looks:

Auto-scroll property

Now we have to connect that checkbox to some ViewModel. We’ll update its XAML definition as:

<CheckBox x:Name="AutoScrollCheckBox"
          IsChecked="{Binding Path=IsAutoScrollEnabled}"
          Content="Auto-scroll"
          HorizontalAlignment="Right"
          VerticalAlignment="Center"/>

With that, we’ve tied a yet-to-be-created variable called IsAutoScrollEnabled to our checkbox. Let’s add it in our MainWindow.xaml.cs class:

private bool m_autoScrollEnabled = false;
public bool IsAutoScrollEnabled
{
    get
    {
        return m_autoScrollEnabled;
    }

    set
    {
        m_autoScrollEnabled = value;
        AddLogEntry(0.0f, "INTERNAL", "Auto-scroll is now " + m_autoScrollEnabled);
    }
}

If you were to do this, it wouldn’t work because we’re trying to bind a variable from a DataContext that’s null by default. To fix that we must do this in the MainWindow.xaml.cs’s constructor:

DataContext = this;

Now it will work as expected.

For now, we’ve used our own log capability to show a message in the window whenever we modify the property. If we were to run it and click on the checkbox several times we’d get something like this:

Scroll to bottom

By default, when a new item is added to the ListView that contains our log messages no scroll is performed. We’d want to know when an item is added to the list so we can perform the scroll ourselves.

Our LogEntries variable (which is an ObservableCollection) has a CollectionChanged event handler we can subscribe to. So, first of all let’s create a method in our MainWindow.xaml.cs to handle the scroll:

private void OnLogEntriesChangedScrollToBottom(object sender, NotifyCollectionChangedEventArgs e)
{
    if (!IsAutoScrollEnabled)
    {
        return;
    }

    if (VisualTreeHelper.GetChildrenCount(LogEntryList) > 0)
    {
        Decorator border = VisualTreeHelper.GetChild(LogEntryList, 0) as Decorator;
        ScrollViewer scrollViewer = (ScrollViewer)VisualTreeHelper.GetChild(border, 0);
        scrollViewer.ScrollToBottom();
    }
}

Basically, we ask the LogEntryList ListView for its children, get the scroll control for the list and ask it to go to the bottom. We can also check the NotifyCollectionChangedEventArgs to know whether it was triggered because an item was added or deleted, if we wanted to.

Now, to tie it with our checkbox, add this:

LogEntries.CollectionChanged += OnLogEntriesChangedScrollToBottom;

We can now, also, refactor IsAutoScrollEnabled with just this:

public bool IsAutoScrollEnabled { get; set; }

Congratulations, you’ve got an auto-scrolling ListView!

Per system filters

So far the new features are great additions, but how about being able to filter messages by their system?

Imagine you’re working on program and there’s some bug you’re tracking down related to the net communication. Wouldn’t it be awesome to filter out everything else and only read the logs for that system? Something easy like clicking on different checkboxes?

However, the goal of this log window is to be used in more than a single project and each one of them may have completely diferent systems. So, how do we do it?

Configuration from the host

Remember our LoggerUI entry point? We’re adding a new method to let the host program tell us which systems it will be using so we can create as many checkboxes. This could be the outline:

public void ConfigureSytems(List<string> systems)
{
    Debug.Assert(m_application != null);

    m_application.Dispatcher.BeginInvoke((Action)delegate
    {
        Debug.Assert(m_application.MainWindow != null);

        (m_application.MainWindow as MainWindow).ConfigureSystems(systems);
    });
}

Of course, MainWindow.ConfigureSystems doesn’t exist, so we must create it. But first, let’s think about what we need at the Window level.

We’re going to have a list of checkboxes, one for each system we’ve received and we want to know whether they are checked to filter the messages. Phew! Let’s digest all that.

ViewModel

We could have a new ViewModel called LogSystem which contains the name of the system (as received from the host) as well as the state of the checkbox we’ll present to the user. Something like:

public class LogSystem
{
    public string Name    { get; set; }
    public bool   Enabled { get; set; }
}

And this way, we can have a List<LogSystem> (or ObservableCollection<LogSystem>) in our MainWindow that contains all of the entries. By having that, we could have MainWindow.ConfigureSystems do this:

public void ConfigureSystems(List<string> systems)
{
    systems.ForEach((system) =>
    {
        LogSystems.Add(new LogSystem
        {
            Name = system,
            Enabled = true
        });
    });
}

Filtering

Remember LogEntries? The List<LogEntry> where we store all of the log messages we receive? Well, we want to filter it now. To do that, we’ll create a new ICollectionView from it.

An ICollectionView lets us sort, filter or group data from a given collection via predicate. This predicate is a function that decides, for each element in the collection, whether it belongs to the filtered data.

By the way, don’t let the View part of the name fool you. This doesn’t have anything to do with visual representation, it’s just the way to call this kind of filtered collection.

So, we’ve got to create it in our MainWindow:

private ICollectionView FilteredLogEntries;

And initialize it:

FilteredLogEntries = CollectionViewSource.GetDefaultView(LogEntries);
FilteredLogEntries.Filter = LogEntriesFilterPredicate;

As you can see, we take LogEntries and create a view of the data. By the way, you may want to transfer the CollectionChanged event to this one so we apply the scroll when this one changes and not the unfiltered one. Then we assign our filter, which looks like this:

private bool LogEntriesFilterPredicate(object item)
{
    LogEntry entry = item as LogEntry;

    // filter out systems
    if(LogSystems.Any(s => s.Name == entry.System && !s.Enabled))
    {
        return false;
    }

    return true;
}

In other words, we keep any LogEntry unless the matching LogSystem is disabled.

This implementation has an unexpected behavior at this point, but let’s continue and discover it later on.

Layout update

We said we’d like to have as many checkboxes as systems so we could filter them to fit our needs. So, we need to add them somewhere. Maybe… to the bottom like this?

For that, and again skipping some of the XML attributes, we have to update our layout to this:

<Window ...>
    <DockPanel>
        <GroupBox ...>
            <CheckBox ... />
        </GroupBox>
        <GroupBox DockPanel.Dock="Bottom"
                  VerticalAlignment="Top"
                  BorderThickness="0">
            <ListView Name="Systems"
                      ScrollViewer.HorizontalScrollBarVisibility="Disabled"
                      BorderThickness="0">
                <ListView.ItemsPanel>
                    <ItemsPanelTemplate>
                        <WrapPanel Orientation="Horizontal"></WrapPanel>
                    </ItemsPanelTemplate>
                </ListView.ItemsPanel>
                <ListView.ItemTemplate>
                    <DataTemplate>
                        <CheckBox Content="{Binding Name}"
                                  IsChecked="{Binding Enabled}">
                        </CheckBox>
                    </DataTemplate>
                </ListView.ItemTemplate>
            </ListView>
        </GroupBox>
        <ListView x:Name="LogEntryList"
                  VerticalAlignment="Stretch"
                  HorizontalAlignment="Stretch">
        </ListView>
    </DockPanel>
</Window>

And remember to set the ItemsSource of the Systems element!

And this is how it would look like:

Impressive, isn’t it? Except it has a bug.

Testing filter

If you asked me, before even executing the log window, What should it do if I disable the TEST system for a while and then enable it? I’d say it would be silent for that while and then all the muted messages would pop at once. Do you mind if we test it now?

First, let it show some messages then disable the TEST system:

After 5 seconds, let’s enable it again.

Can you spot the issues?

The first screenshot shows how we wanted to filter out the messages with the TEST system but the ones before we disabled it are still there.

The second one shows how the messages between timestamps 3651 and 8797 are gone! It’s like they weren’t registered at all! Why is that?

If you remember, we created an ICollectionView from the LogEntries list. When we created it, the filter was executed and it’s executed again when a new LogEntry is added to the original LogEntries collection.

When we disabled the TEST system, new items didn’t fulfill the predicate but we didn’t re-evaluate it for the existent ones. When we enabled it again, only the new ones are evaluated and not the ones that were discarded while it was disabled (although they were still stored in the LogEntries list).

So, how do we fix this issue?

INotifyPropertyChanged

We’ve seen how our LogSystem.Enabled property was modified when we interacted with the checkbox. Our goal is to re-evaluate the filter whenever this happens so it’s applied to the whole LogEntries.

To do this we need to allow whoever is interested to listen to changes in our Enabled property. WPF has an interface called INotifyPropertyChanged that lets us execute an event whenever this happens. We’d have to refactor our LogSystem class as:

public class LogSystem : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    public string Name { get; set; }

    private bool m_enabled;
    public bool Enabled
    {
        get
        {
            return m_enabled;
        }

        set
        {
            m_enabled = value;
            if(PropertyChanged != null)
            {
                PropertyChanged(this, new PropertyChangedEventArgs("Enabled"));
            }
        }
    }
}

When inheriting from this interface we have a new event handler called PropertyChanged that lets others listen to the changes we decide to trigger.

Remember our MainWindow.ConfigureSystems method where we created our LogSystem entries? That’s where we’ll subscribe to changes.

public void ConfigureSystems(List<string> systems)
{
    systems.ForEach((system) =>
    {
        LogSystem entry = new LogSystem
        {
            Name = system,
            Enabled = true
        };

        entry.PropertyChanged += OnSystemEnableChanged;

        LogSystems.Add(entry);
    });
}

The new OnSystemEnableChanged would look like this:

private void OnSystemEnableChanged(object sender, PropertyChangedEventArgs args)
{
    FilteredLogEntries.Refresh();
}

Refresh recreates the view by re-evaluating the predicate against the source collection. This way, when we disable the TEST system we’ll have an empty list and when we re-enable it we’ll have all items even when we weren’t seeing them.

Great job! Now you’ve got filters by system in an auto-scrolling ListView!

We’ve gone through some useful functionalities for our window, but we’re missing an interesting one: log levels with colors!

Mind joining me in the next post of the series to complete it?

Thanks for reading!

Log window from scratch: from standalone to class library

2018-06-15T00:00:00+00:00

Hello again! I was waiting for you to continue with our Log Window!

In the last post we created the basic UI and functionality. However, that window does little by itself so we want to convert it into a component we can plug into other projects. That way, we could focus on building a nice project with the help of a logging window!

Grabbed a drink? Then we’re ready to start!

The host program

We said we wanted to use the window from another project, right? Why don’t we start building that first?

Let’s add a new C# project into our solution. This time, we don’t want a WPF Application but a Console Application. Let’s give it a meaningful name… like HostProgram.

We’ll have a new project with a new autogenerated Program.cs file. Let’s keep it and modify its Main to something familiar:

static void Main(string[] args)
{
    Console.WriteLine("Hello, world!\n");
}

Now, set this project as the startup project: one way is to right-click on the project then select Set as StartUp Project. Then run the solution and you’ll have the expected message in the console. Nothing new here.

What if we wanted to use the LogEntry class we created in our LogWindowUI project? Can we do this?

static void Main(string[] args)
{
    LogEntry entry = new LogEntry
    {
        Timestamp = 0.0f,
        System = "TEST",
        Message = "Sample message!"
    };

    Console.WriteLine("[{0}][{1}] {2}",
        entry.Timestamp, entry.System, entry.Message);
}

Unfortunately we can’t… yet.

Referencing projects

For our HostProgram to be able to use classes from the LogWindowUI project we need to connect them somehow. To do so, expand the HostProgram project and right-click References, select Add Reference.... Now enable the checkbox next to the LogWindowUI project and accept the dialog.

Our code still doesn’t compile, but now we can add this line:

using LogWindowUI;

Because, if you remember, our LogEntry class was defined inside a namespace called LogWindowUI.

We can happily run our solution now and the formatted log message will show up in the console. We’re getting there!

Breaking it up

Okay, we’ve connected our projects but our LogWindowUI one is still a Windows Application and we want to make it a Class Library. We’ll do that now, but first you should know everything will be broken until we complete some steps.

Start by opening the Properties page in the project (one way is double-clicking the Properties entry when you expand the project). Select Class Library from the Output type control, then save.

You will now see several errors in the Error List panel. The first one says:

Library project file cannot specify ApplicationDefinition element.

Removing App.xaml

Do you remember, from previous post, that the startup of the UI felt like Magic because we apparently never said where it should start? Remember the two concepts we discussed: Application and Window?

One of the files, App.xaml, defines the entry point and which Window to display when we launch the program. Because we’ll be using this project as a Class Library, the host program will be responsible of managing the life cycle itself. We’ll just provide tools for it to work. So, delete App.xaml and the error will go away.

Except not! It looks like it has deleted App.xaml.cs as well and we don’t have an Application anymore!

Creating App.cs

Let’s add a new class called App in our LogWindowUI project with this code, pretty similar to what App.xaml.cs had:

namespace LogWindowUI
{
    class App : Application
    {
    }
}

With this, all errors will go away and we can run our console-based hello log. Impressive :)

But, how do we connect them now?

Showing window from host program

So far we’ve gained access to the classes in the LogWindowUI project from the HostProgram one. However, we’re left with a sad log message in the console and no WPF window.

To keep things clean, we should expose some class that lets us create the window, send it messages and close it when necessary instead of dealing with the internal classes from the outside. We only want to have one window at the same time, so… guess which design pattern we’ll be using?

Singleton pattern

You may have heard of the Singleton pattern before. It allows us to restrict the instantiation of a class to a single object. It’s used when it doesn’t make sense to have several instances of one class, commonly when you’re defining a manager of some sorts or a wrapper to an underlying system which should have a single entry point.

Bob Nystrom (@munificentbob), in his Game Programming Patterns, has an awesome chapter about the pattern so go and read it if you want to know the good and bad parts about it.

A simple C# implementation of the pattern is this one:

public sealed class LoggerUI
{
    private static LoggerUI m_instance = null;
    public static LoggerUI Instance
    {
        get
        {
            if(m_instance == null)
            {
            	m_instance = new LoggerUI();
            }
            return m_instance;
        }
    }

    private LoggerUI()
    {
    }
}

However, this lazy initialization (the instance isn’t created until the first piece of code accesses it by using the Instance property) takes life cycle control from us (and isn’t thread safe). We’d want to explicitly control when the instance is created and when it’s destroyed. Let’s make it so:

public sealed class LoggerUI
{
    private static LoggerUI m_instance = null;
    public static LoggerUI Instance
    {
        get
        {
            Debug.Assert(m_instance != null, "LoggerUI not initialized");
            return m_instance;
        }
    }

    private LoggerUI()
    {
    }

    public static void Initialize()
    {
        Debug.Assert(m_instance == null, "LoggerUI already initialized");
        m_instance = new LoggerUI();
    }

    public static void Destroy()
    {
    	Debug.Assert(m_instance != null, "LoggerUI already destroyed!");
        m_instance = null;
    }
}

This way, we control the life cycle ourselves. It isn’t thread safe either, but we’ll be sure to Initialize and Destroy in a single-thread part of the program. Let’s also add a new method to test it:

public void Add(string message)
{
    Console.WriteLine(message);
}

Calling from host

For now, our host program just has a Main function from which we’ll call everything:

static void Main(string[] args)
{
    // intialization step
    LoggerUI.Initialize();

    // [...] some other code in our program

    LoggerUI.Instance.Add("Hello, world!");

    // [...] some other code in our program

    // teardown step
    LoggerUI.Destroy();
}

And it works as expected, yay! Now, let’s show the window again!

STA, Application and Window

Remember the App.xaml file we had before we broke everything? If we stripped it a bit we’d have this:

<Application x:Class="LogWindowUI.App"
             StartupUri="MainWindow.xaml">
</Application>

Which is telling the runtime to use the LogWindowUI.App class as the Application and then display the Window defined at MainWindow.xaml (whose x:Class attribute points to MainWindow.xaml.cs).

So it looks like we need an Application that displays a Window. Okay, what else?
In the previous post we mentioned WPF uses the Single Thread Apartment model: any object created within a thread can only be modified from that thread (the UI Thread). Guess we need one of those too.

We’ve defined our LoggerUI singleton class as the entry point to our WPF library. We could have the Application as a member of LoggerUI and tie its creation to the creation of the singleton itself. Something like this:

private App m_application = null;

private LoggerUI()
{
    Thread t = new Thread(() =>
    {
        m_application = new App();
        m_application.Run(new MainWindow());
    });
    t.SetApartmentState(ApartmentState.STA);
    t.Start();
}

Basically, just what we said we’d do. But there’s a catch.

When we run this code, it’s not guaranteed the Application and its Window are running after we call LoggerUI.Initialize() (which ends up calling the constructor and thus this code). If we were to start logging stuff right after calling Initialize it could fail because m_application could be uninitialized. Ah, the joy of multithreading.

One way of solving this issue is to wait for the Application to be ready. After all, it’s part of the initialization process. We can use an AutoResetEvent to wait in our main thread until the Application and Window are running. We could rewrite the constructor with this code:

private LoggerUI()
{
    AutoResetEvent windowCreatedEvent = new AutoResetEvent(false);
    Thread t = new Thread(() =>
    {
        m_application = new App();
        MainWindow window = new MainWindow();

        m_application.MainWindow = window;
        m_application.MainWindow.Show();

        // notify they are created before we block this thread
        windowCreatedEvent.Set();

        m_application.Run();
    });
    t.SetApartmentState(ApartmentState.STA);
    t.Start();

    // wait until the application and window are created
    windowCreatedEvent.WaitOne();
}

Note how we’ve separated the Window creation from the Application’s Run method. When you create a Window it starts hidden, so we must call Show ourselves or let Run do it for us. Oh, and Run blocks until the Window is closed, so we must signal the event before doing it. The Application is already created by then so we can start manipulating it.

If we were to run the program now we’d see something like this:

All that’s left is sending messages from the host program to our WPF library instead of mocking them!

Connecting logs

As you can see in the previous picture, we had a way of adding messages to the window. For testing purposes we did that in the MainWindow’s constructor. Let’s delete that code now, so there’s nothing sending messages.

Now, in the LoggerUI singleton class, let’s update the Add method:

public void Add(float timestamp, string system, string message)
{
    Debug.Assert(m_application != null);

    // add it to the window via UI thread
    m_application.Dispatcher.BeginInvoke((Action)delegate
    {
        // window can be closed already
        if (m_application.MainWindow == null)
        {
            return;
        }

        (m_application.MainWindow as MainWindow).AddLogEntry(timestamp, system, message);
    });
}

Did you notice the Dispatcher.BeginInvoke thing we mentioned in the previous post?

Also, create MainWindow.AddLogEntry like this:

public void AddLogEntry(float timestamp, string system, string message)
{
    LogEntries.Add(new LogEntry {
        Timestamp = timestamp,
        System = system,
        Message = message
    });
}

Our current code won’t compile if we do this, because in our Main function we’re still using the old version of Add. Let’s fix it:

LoggerUI.Instance.Add(0.0f, "TEST", "Hello, world!");

Run it and we have this:

Or we could do this as well:

for(int i = 0; i < 100; ++i)
{
    LoggerUI.Instance.Add(0.0f, "TEST", "Hello, world!");
    Thread.Sleep(100);
}

Yay! It works! Looks like we did it, awesome job!

In this post we’ve managed to convert our WPF project into a Class Library we can plug into other projects and log whatever we want from that project. We’ve also defined a Singleton entry point for the library and created the UI Thread in which the Application and Window live.

In the next entry we’ll work on adding some extra functionality to the window, like colors, filters and auto-scroll.

See you then!

Log window from scratch: basic window

2018-06-13T00:00:00+00:00

Welcome back, dear reader!

In this post we’ll create a basic standalone log window in WPF that we’ll use as the foundation of our project. I recommend you checking out the previous entry in the series if you want to know the motivation or the road map.

We’ll end up with this, after the post:

Creating the base project

We’re using WPF, so start up your Visual Studio and create a New project.

Under Templates find Visual C# and then select WPF Application. Choose a nice name for your brand new project (and solution) and accept the wizard.

After it finishes loading up, you’ll have a solution with a project and several files. These are the basics of a Windows Presentation Foundation (WPF) project. There are two basic concepts:

Application: the WPF program itself, the entry point to the framework.
Window: a piece of UI where controls are displayed so user can interact with the program.

In the project you have three files related to the Application:

App.config: general configuration, metadata and settings for the application (database connection strings, .NET version, …).
App.xaml: declares which Window to display as it starts and extra data resources.
App.xaml.cs: C# class for this application.

There are also two files related to the Window:

MainWindow.xaml: declares the controls for the window, their properties and some behavior.
MainWindow.xaml.cs: C# class for this Window, which will hold its custom logic.

At first, when I started learning the little bit of WPF I know now, it felt like Magic. Where is the entry point defined? There’s no Main function! How does it know which window to display? Hint: it all starts in the Properties page and ends up in the StartupUri of App.xaml.

For now, we’ll just work with these default files and configurations as a standalone WPF project. In the next post, we’ll convert it to a Class Library and will learn how the Application-Window relationship works.

Basic UI controls

Okay, so we have the default WPF stuff. Let’s start creating our UI!

If you remember, the bare minimum we wanted to have a tabular display of the log data. Maybe with three columns:

Timestamp: can be any kind of timestamp we want from seconds since startup to full date timestamp.
System/tag: each message can be tagged with a key, like the system it comes from (i.e. net or render).
Message: the log data itself.

Let’s open the MainWindow.xaml file now. The Designer will open for us showing two panels: one with the graphical representation of our Window and one with the XAML markup. It looks something like this (some names will differ):

<Window x:Class="LogWindowUI.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:LogWindowUI"
        mc:Ignorable="d"
        Title="MainWindow" Height="350" Width="525">
    <Grid>
        
    </Grid>
</Window>

Add a ListView into the XAML panel with this markup:

<Window ...>
    <Grid>
        <ListView x:Name="LogEntryList">
            <ListView.View>
                <GridView>
                    <GridViewColumn Header="Timestamp" Width="70"></GridViewColumn>
                    <GridViewColumn Header="System" Width="70"></GridViewColumn>
                    <GridViewColumn Header="Message" Width="440"></GridViewColumn>
                </GridView>
            </ListView.View>
        </ListView>
    </Grid>
</Window>

And this is how it looks when we run the program:

Alright, we’re on track!

MVVM pattern

So far we’ve just defined how we want our window to look like but it doesn’t do anything just yet. We need to understand how WPF handles data, first.

Although WPF lets you tie your UI and your logic in a tightly-coupled way (code knows which control to update when something happens, control knows which data to update, …), the MVVM way is preferred.

MVVM stands for Model-View-ViewModel and it’s an architectural pattern that helps separate the UI representation (the View) from the program logic and its data (the Model). The ViewModel is kind of a converter from the Model to the View (and handles the presentation logic).

In other words, the Model is connected with the ViewModel, which is connected to the View. Whenever the Model changes, the ViewModel notices and notifies the View to update accordingly. When the user interacts with the View, the ViewModel is modified which, in turn, ends up in an update of the Model.

How does this affect us? Well, first of all, let’s understand which one of them maps to which concept in our program.

Model

For us, each log entry is a Model. It has a timestamp, a system and a message (remember last screenshot?).

In other words, this is our Model:

public class LogEntry
{
    public float  Timestamp { get; set; }
    public string System    { get; set; }
    public string Message   { get; set; }
}

View

If you have a look at the previous XAML code we’ve listed, you can see we created a ListView called LogEntryList. Suspicious names, huh? If we take last screenshot and only keep our View, we’d have this:

ViewModel

Our LogEntryList control takes any IEnumerable object as its ItemsSource and displays it. The View relies on the ViewModel notifying of a data change to update its visual representation. So, if we used a List<LogEntry> we’d see nothing when we add a new item: it doesn’t notify anyone.

WPF comes with an interesting collection called ObservableCollection<T>. It knows how to notify of a data change in the collection itself (an element is added or removed). So, open MainWindow.xaml.cs and add this property in the MainWindow class:

public ObservableCollection<LogEntry> LogEntries;

In this case, we could say that the LogEntry Model is also acting as a ViewModel because we’re using it directly as part of the ObservableCollection. However, we could have a hard separation between the Model and the ViewModel if we were doing more than just showing data (like having extra properties only relevant to the View or a data conversion from the Model to the View).

In future entries of the series we’ll see concrete examples of custom ViewModel.

Displaying logs

Okay, now that we know what’s what, let’s get some messages on screen!

First of all, let’s hardcode a message we’ll display. Open the MainWindow constructor and make it this one:

public MainWindow()
{
    InitializeComponent();

    LogEntries = new ObservableCollection<LogEntry>();

    // tie View with ViewModel
    LogEntryList.ItemsSource = LogEntries;

    // add test data
    LogEntries.Add(new LogEntry
    {
        Timestamp = 0.1f,
        System = "TEST",
        Message = "Hello, codingScars!"
    });
}

When we run the program this is what’s displayed:

That’s because we haven’t told the View how to display each property of an individual entry. To do so, let’s modify the MainWindow.xaml file and update the ListView control with this code:

<ListView x:Name="LogEntryList">
    <ListView.View>
        <GridView>
            <GridViewColumn Header="Timestamp" Width="70"
                            DisplayMemberBinding="{Binding Timestamp}"></GridViewColumn>
            <GridViewColumn Header="System" Width="70"
                            DisplayMemberBinding="{Binding System}"></GridViewColumn>
            <GridViewColumn Header="Message" Width="440"
                            DisplayMemberBinding="{Binding Message}"></GridViewColumn>
        </GridView>
    </ListView.View>
</ListView>

The DisplayMemberBinding attribute defines which data to display in a column. With the {Binding Message} value, we’re telling WPF to look for a Property called Message in the LogEntry object. Thank Reflection for that! Please note, {Binding ...} is used to connect the View with the ViewModel and it can be done with properties other than DisplayMemberBinding.

Run it again and it will show this:

Yay! We’ve got it!

Live update

Okay, okay, we’ve displayed some text, but it’s not displaying anything live. Why don’t we do that now?

First of all, remove the code that adds an entry to the LogEntries collection. Let’s now create a new Thread that will be adding entries to the LogEntries collection.

This is the code:

// add test data
new Thread(() =>
{
    LogEntries.Add(new LogEntry
    {
        Timestamp = 0.1f,
        System = "TEST",
        Message = "Sample message!"
    });
}).Start();

We run it and… boom! The program crashes! Why is that? What does this error mean?

This type of CollectionView does not support changes to its SourceCollection from a thread different from the Dispatcher thread

STA

WPF uses the Single Thread Apartment model. This means that, any object that’s created within a thread can only be modified by that thread. The Application is running in what we call the UI Thread, which is the main one in our case. Because we’ve created our own thread, we can’t modify LogEntries directly: this new thread doesn’t own the collection.

To make it work, let’s change the code to this one:

// add test data
new Thread(() =>
{
    App.Current.Dispatcher.BeginInvoke((Action)delegate
    {
        LogEntries.Add(new LogEntry
        {
            Timestamp = 0.1f,
            System = "TEST",
            Message = "Sample message!"
        });
    });
}).Start();

Run it again and… it works!

Dispatcher

Whenever we want some code to be executed in the UI Thread we can use the Dispatcher object from the Application instance to queue the actions we want to perform.

There are two important methods to queue actions: BeginInvoke is asynchronous and Invoke is synchronous.

Final test

So, now that we know how it works, let’s have a more complete example!

// add test data
new Thread(() =>
{
    long ticks = DateTime.Now.Ticks;
    Random random = new Random();

    for(int i = 0; i < 100; ++i)
    {
        App.Current.Dispatcher.BeginInvoke((Action)delegate
        {
            LogEntries.Add(new LogEntry
            {
                Timestamp = (float) (DateTime.Now.Ticks - ticks) / TimeSpan.TicksPerSecond,
                System = "TEST",
                Message = "Sample message!"
            });
        });

        Thread.Sleep(random.Next() % 100 + 50);
    }
}).Start();

Running it will show something like this:

Phew! Not bad! Good job :)

Through the post we’ve created the foundation of our log window from scratch and learned about the MVVM pattern. In the next post we’ll take this standalone WPF program and make it a Class Library that we can use from a host program.

Thank you for reading! See you then!