Silviu-Marius Ardelean's blog

a software engineer's web log

The First Hackathon Experience #HackTM2016

It have passed approximately two months from my first hackathon experience, the #HackTM2016 from Timisoara. This delay I’m publishing this article is mostly because of the release period to the job and other personal stuff I had to do.

This experience was a reminder of my high school programming competitions where I have attended with different applications written in the already ancient Turbo Pascal 7.0 or Borland C++ 3.1. At that moment, probably because of my high school informatics great teachers, the competition spirit between colleagues was so intense and we have competed each other in creating applications within the local competitions and in other cities from the country.

This year, the hackathon competition from Timisoara was between 20 – 22 May 2016 to the UPT Restaurant, Timisoara and from my point of view it was a great event. The event had categories such eHEALTH, Robots, Smart City, GameDev, IoT, Education, Fintech and Open but no Automotive category as I expected according to previous pitching sessions.

Before the effective event I have attended in two pitching sessions seeing different challenging proposals in few domains. The most appealing were some ideas of the ADAS team from Continental, a team I was apart between 2006-2008 (Siemens VDO department, video camera projects at that time).

Even if I had no team, because of curiosity, I bought the ticket and I went to the hackathon pitching session, trying to figure a team and to decide a project to deal with.

The Continental ADAS team came with hardware and software support how to hack their platform.

We were able to find a small team, first by two persons and later for a period five persons. Unfortunately those last three persons left us while realizing that the project is not what they have imagined initially, that other projects look more challenging or that their knowledge were not matching with what it was required for our project.
So, I installed on my laptop some software used internally by the ADAS team and we took a ride to record real traffic data. Once we had this data, the effective programming for grabbing it can be done in office conditions.

Driving to grab ADAS data

The idea

Having the radar and camera information, GPS data and vehicle dynamics, we were thinking is that in the near future, even before self-driving cars on the common roads, these information might be sent in cloud to be used by the traffic management solutions in order to be optimized the traffic. Even these days, in my city, Timisoara, such traffic management solution is implemented, but is not based on cars internal information. Maybe, this idea will not be applied.

What we have done – the effective experience

So, facing with this challenge, me and my teammate Nikola Kolevski, a Serbian nice guy, have started the work on Friday evening. We have met on that pitching meeting and we had a great cooperation. I was the back-end guy and he the man from the cloud. Because we “spoke” different languages, me C++ and he Python, we have decided to speak the REST way.

During our job we have improvised the “project management” with a Trello board. Of course, we used GitHub for source control.

What I effectively had to do it was to inject some code within a .DLL that was loaded into a Continental application and sent the ADAS information in cloud. Nicola was the guy that received and collected the data. Unfortunately, even if we tried few times, we found no front-end available teammate, just some with slices of time in terms of availability.

ADAS AE-RO HackTM concept

I decided to use the benefits of asynchronous programming and I used the Casablanca REST API. But because of the Continental’s Visual Studio 2012 project constraints (!!!), during the Friday evening I faced up with the challenge to find and adapt a Casablanca library older version to the project. The latest Casablanca’s versions are available for VS 2013 and VS 2015 only. Thanks to NuGet tool, I finally managed to get and use the 1.10 version.

On Saturday we managed the effective work, facing with some challenges related to the TCP/IP communication between our applications because of some Python server configuration, but finally our applications were able to talk each other via REST services, in the night and I have tweeted.

After a sleep break, next morning we did some last code adjustments and being time constrained we tried to improvised a frontend. Also, we had a second trip with the Continental’s Mercedes car to test what we did, using a 4G network.

As usual for a hackathon, everything was on the run with adrenaline, so quite nice! At the end of the hackathon we had to prepare for the hackathon jury’s visit and later for presentation because we have qualified in the first three teams in the Robots section. Yeah, we were included there because the Automotive category did not exist, but it was fine. 🙂

The truth is that the Continental was the only automotive represented company even if there are many such companies in Timisoara, but it seems they are not interest in such events.

Other interesting things from #HackTM2016

Attending to #HackTM2016 was a great experience, I have seen many interesting projects but from far the most exciting one was the Symme 3D Printer, a local start-up.

Conclusions

In a internet of things world our based idea might connect the cars ADAS information with intelligent management future systems to improve the traffic flow in big cities.

It is obvious but I want to underline: if you want to have success in hackathon, try having a core team before event. Otherwise, you might just have fun coding but not ending the prototype.  Strategies of being efficient would be a great asset.

Meeting new people and trying to do something from the scratch in a limited time is a very cool thing even if you don’t have time to write optimized and tested code. Also, you might learn a lot of new things.

Definitely, I will repeat this experience in the future!

Share

The Chameleon Pathnames

The title might be as well “When the pathname is not what it has to be”.

The experience of developing plugins for Adobe Acrobat/Reader reserved me different surprises, surprises that made the task more challenging. One of the biggest surprise I had was the impact of the Adobe’s Cloud idea over the Acrobat’ API within Acrobat products. Their feature idea is to keep all the already opened documents within their Cloud in order to make them available to different devices you’re interacting with.

In my case, having interactions with external non-Adobe’s applications, the things complicated when trying to get the file pathname. This option is coming enabled by default.

This is how the Acrobat.com Cloud looks like within Acrobat products This is how the Acrobat.com Cloud looks like within Acrobat products

Usually, when we are thinking to files path we expect to have something similar to GetFullPathName(). But according to Acrobat SDK’s concept: not every file opened into Acrobat/Reader has to be a local disk file. It may be associated with a stream, a network file, etc.

The reason why I was looking to get the correct file path is that my plugin and others are connecting to a system that expects the local or network file path. So it was needed to find a way to get an usual file path.

The challenge I am talking about has reproduced with an Adobe.com environment activated, having such a file already synchronized in the Acrobat.com cloud by using:

acrobat.com_path_functions

But both API’s functions return proper values with non-Cloud files. With a local filename not already uploaded within Acrobat.com I got the correct file path with both functions.

So the workaround I was thinking invokes the next steps:

1. Get the file path using ASFileSysDIPathFromPath(). In case your project is a Unicode project don’t forget that the returned type is a char* and you’ll need to encode it to the proper Unicode (UTF-8 in my case).

2. Check if it is a cloud path (starts with Acrobat.com:).

where

3. Save the file content into a temporary file (ex. C:\Users\Silviu.Ardelean\AppData\Local\Temp\)

where

4. Provided the temporary file path to the proxy module that expect it to interact with my system.

5. Clean/delete the temporary files on plugin uploading – PluginUnload() callback.

Additional comments
In case your plugins will interact with external non-Adobe’s application most probably you’ll have to do different tricks. Because of the way the Acrobat SDK is designed, without direct support for wchar_t and std::wstring you will need do make different conversions and encoding/decoding (ex. the ATL macros CW2A, CA2W(), functions such MultiByteToWideChar() on Windows, etc).

If you don’t have to interact with external non-Adobe’s applications be confident with Acrobat’s SDK types and data structures. In this way you’ll avoid such conversions.

Share

Experiences with Adobe Acrobat/Reader Plug-ins

box_adobe_150x150I wrote this document after a challenging experience I had recently within an Adobe Acrobat/Reader plugin creation. Even if the Adobe’s SDK it’s nicely documented within PDF files, one of the major reason that determined me to write this article was the frustration I got sometimes when, for instance, trying to see “why the plugin was not loading into Acrobat/Reader” and the Google’s engine provided me a lot of references such “why the Reader plugin is not loading into a browser”. Also, the search functionality from Adobe’s forum didn’t helps me too much. I hope to help others by clarifying some challenges might meet a developer at begin of creation such kind of plugin.
Adobe has two products for .PDFs file handling: the freeware Reader capable for reading only and the Acrobat for read, write and effective .PDFs creation activities. Both Acrobat and Reader use the same SDK but the Reader APIs is a subset of those available for Acrobat (obvious).
There are three types of plugins: regular plugins, reader-enabled plug-ins and certified plug-ins.

General considerations

Plug-ins for Acrobat can be developed and distributed freely and no license is required from Adobe. The payment exception appears in case of DRM agreement which includes a $50,000 annual fee and a 5.5% revenue royalty. Adobe consider to apply digital rights management (DRM) in case the developed plugins functionality invoke “encrypting a PDF file or controlling access to a PDF file, then it is DRM. Also, if you add any functionality to the security settings of Adobe Acrobat (…). If your plug-in is required for someone to access the PDF file, then we would consider it to have DRM functionality”.

Only plug-ins that are shipped as part of Acrobat and Reader can be ‘certified’. This is so that if users wish to run Acrobat or Reader without any 3rd party plug-ins, they can do this easily by using the ‘Certified plug-ins only’ check box in the preferences.

Adobe maintains a registry of four-character prefixes for each company that develops extensions for its own products. The new companies that intend to develop such plugins should contact Acrobat Developer Support to obtain a four-character prefix to be used. Adobe’s prefix is ADBE or ACRO. This prefix is needed to be use with various elements as well as private data that it writes into PDF documents.

For Adobe Reader the plugin needs a special macro to be defined into project settings READER_PLUGIN. By defining it it’s easy to identify in case you’re calling an Acrobat only specific SDK function because it causes compiler errors.

The First challenges

After downloading the SDK the first instinct was trying the project samples. Once with this step appeared the first annoying situation: I loaded all.sln solution into Visual Studio I have been observed that whatever project I built and deployed into the Reader “plug_ins” subfolder I was not able to see them into Adobe Reader. The “plug_ins” subfolder or one more subfolder level down is the place you have to deploy the built plugins. These plugins are DLLs with an .API extension. The confusion has amplified because by downloading and installing the FileOpen WebPublisher Client plugin I observed that the plugin was running perfectly and I saw it even into Help – Adobe Third Part Plug-ins menu.

But deploying such plugins in the Adobe Acrobat “plug_ins” folder were up and running. I started reading the Developing Plug-ins and Application for Adobe Reader I followed the “why a plug-in might not load” founded steps but no solution for Acrobat Reader. Just in case I unchecked the “Use only certified plug-ins” Reader’s setting and nothing (‘Certified plug-ins only’ = Edit > Preference: Application Startup: Use only certified plug-ins (unchecked)).

Trying to debug over the plugin source code by attaching to Reader project or by starting the debug with Reader application didn’t help me more.

Later, after some challenges, I found out that the key point of understanding why the SDK sample DLLs were not loading into Adobe Reader it was that the plugins for Reader need to be signed before being deployed into “\Program Files\Adobe\Acrobat\plug_ins” directory. Such information is not present into that “why is not loadingmanual list.

How to sign the plug-in for Adobe Reader

As I mentioned earlier the plug-in in Reader must be signed by using a certificate provided by Adobe. It is strongly recommend that you make the application for a key at the beginning of the development process, since the application can be denied in case the plugin functionality is not in accordance with Adobe’s business goals. Also, this ensures that your agreement is in place when you are ready to build the Reader version of the plug-in. If the key is approved, the developer must build public key and pair key files using a tool in the Acrobat SDK.

In the moment somebody wants to develop a plug-in for Acrobat Reader has to fulfill an integration form, not before creating an Adobe ID. According to Adobe the approval process might take some time (up to two weeks). The application should be filled out completely and your responses will be used to determine your eligibility. If you are building a DRM-based Adobe Reader plug-in, we recommend you send an email to with details of your request so that we can guide you through the application process.

Generates the public and private pair keys by the Makekeys tool:

The size of the public key should be 98 bytes. The size of the public and private key pair should be 451 bytes. The size of the returned encrypted key should be 554 bytes. Save this .key generated files into a proper location cause later might be useful including it into your project.This tool is located into your SDK: ex. sdk110_v1\PluginSupport\Tools\Reader-enabling Tools\win.

Submit the new created Public Key file and the fulfilled form document to and wait to get the digital certificate. This will be a RIKLA-DigCert.rc file.  In case you will receive approval from Adobe there are several more steps you need to follow to receive your Reader Integrated Key for your plug-in. The Key arrives as a digital certificate. Once this is done the plug-in will load into Reader. Note that if the plug-in is recompiled the plug-in must again be signed (the same certificate and key-pair files can be used).

Once you get the digital certificate file, you should sign the fresh built plugin, before deploying it into Reader’s plug_ins folder.

Here, because of using SDK 11 I got some confusion because of the steps described in the “enabling the plug-in for Acrobat Reader” section, according to apps documentation guide. They are talking about Makecmd32.exe tool, some API_ENCRYPTED_GIGEST and API_DIGITAL_CERTIFATE IDs, etc. But the SDK 11 has no Makemd32.exe tool coming with. This tool can be downloaded within other RIKLATools.zip file but I preferred following the actual SDK 11 documentation especially because it has other signing approach. Instead of Makemd32.exe I had to use SignPlugin.exe (into SDK docs: Plug-ins and Applications > Developing Plug-ins and Applications > Creating an Adobe Reader Plug-In > Developing and enabling an Adobe Reader plug-in > Enabling the plug-in for Adobe Reader).

Plug-In Structure

The Acrobat/Reader applications have few steps approach for plugins: initialization + plugin, handshaking, exporting and importing HFTs and unloading the plugin, implemented as callbacks. A minimum operation that a plug-in must implement is PluginInit() callback function.

The plugin life cycle into Adobe Acrobat/Reader invokes next steps:

  • At startup search into its plugin directory (plug_ins). It looks in the .API files for the exported PlugInMain, it loads the plugin by invoking the LoadLibrary and call the function pointed by the symbol of PlugInMain.
  • For each detected plugin (.API) it attempts loading the file. If the plugin is successfully loaded the Reader/Acrobat invokes routines from PIMain.c and completes the handshaking process.
  • Invokes callbacks in the next order:

PIHandshake

PluginExportHFTs

PluginImportReplaceAndRegister

PluginInit

  • Before closing Reader/Acrobat the PluginUnload callback function it’s executed. That’s the proper place to release the allocated resources.

In the initialization phase the plugin hooks into Acrobat’s user interface by adding menu items, toolbars, etc. The unload procedure should free any memory the plug-in allocated and remove any user interface changes it made.

Handshaking is also one of most important step. The application performs checking with each plugin before opening it. It is the step where a plugin for Adobe Reader it is tested before loading. During this operation the plug-in specifies the name, some initialization procedures, signature test and optional an unload procedure if is needed. In case the signed test fails the loading process of that plugin is stopping.

How to create a plugin

Even if the Acrobat SDK allows creating plug-ins for OS platforms out of Windows (MAC, Unix/Linux) without too many differences (most because of configuration and used tools), I will describe down some details for plugin development on Windows platform.

Download the latest Acrobat SDK and unzip in a preferred location. Create an environment variable for AcroSDKPIDir that contained the SDK content.

Running Visual Studio “as administrator” it’s a good idea in order to be able to succeed the write into Adobe’s plug_ins folder. In order to establish an easier debug and deploy process I preferred to add two additional environment variable AcroPluginsDir containing the Acrobat plugins files and ReaderPluginsDir for Reader plugins files.

Having these environment variable set into your OS you can start the effective plugin creation.

According to Acrobat SDK you can start from an existing sample so called Starter project or you can start from an empty DLL project. The first version allows you having a fast up and running own plugin by just adjusting the files name and starting to apply the business logic.

In case you choose the clean approach you need to add paths to the SDK header files into C/C++ > General > Additional Include Directories as for instance:

This will be needed for instance to easily include “PIHeaders.h” file.
Add next preprocessors definitions into project settings: WIN_PLATFORM, WIN_ENV and READER_PLUGIN (C/C++ > Preprocessor > Preprocessor Definitions).
Include PIMain.c file into your project. This file is located into your Acrobat SDK path. In my case it is:

Add the standard Acrobat callbacks functions prototype into other .cpp file (functions invoked into plugin structure topic) and start the business logic implementation. Here you can inspire from the content of StarterInit.cpp file (Starter sample project). In case you want to add some menu, toolbar or other UI items these should be added into PluginInit() function.

The PlugInMain() function is the entry point into such plugins and it’s needed to add the export flag to PlugInMain() function via project settings:

Without this setting you will get a big surprise even if at the very first point of view the built plugin is signed and the DllMain() is accessed into a debugging flow. But none of the callbacks functions without this export.

In order to automatize the process for plugin build and deployment you might added some Post Build Event commands:

Conclusions: In my opinion, the Acrobat SDK it’s nicely designed but even if there are a lot of PDF references, somehow it doesn’t have the best online structured content, causing users to waste enough time to match all the pieces. Maybe because of complexity and flexibility that exposes it’s not very easy to find complete clean references.

Share

Getting Table’s indexes experiences – workaround

Trying to get table indexes information in SQL Server 2012 I identified a strange situation within a specific method that I was using so long but it was not acting as expected in one situation.

The way of getting indexes information using the ODBC C API into that old and inherited method looks like:

Usually, I got the right information about indexes but in one situation I encounter a strange behavior. It’s about having a clustered index into a scenario.

I have a table that contains two indexes referenced to some fields: IndexField_1 and IndexField_3 mapped over int, NULL fields. When IndexField_1 is Non-Unique, Non-Clustered and IndexField_3 is Clustered index I get the right information.
But if the index IndexField_1 is Clustered and the IndexField_3 is Non-Unique, Non-Clustered I get no information about IndexField_1 index (eg. szIdxName and szIdxColName are “” and their length is -1 that means SQL_NULL_DATA). Within while loop, with the next iteration I get correct information about the second index IndexField_3.

Because SQLExtendedFetch() is deprecated I tried using SQLFetchScroll() but the behavior is the same from my interest point of view.

I was not sure whether the problem is with SQLStatistics, the bindings or SQLFetchScroll (they all always return SQL_SUCCESS). It looks such a problem with the driver when the first index is clustered.
According to SQLStatistics documentation if my swType parameter is SQL_TABLE_STAT I have no information for index or field. But for this scenario I had no indexes of combined fields.
For the good scenario I observed that my while loop had 3 iterations including one of having swType = SQL_TABLE_STAT without information in szIdxName. But for the bad scenario the loop had only 2 iteration. So it looks like SQLExtendedFetch() is not getting the last one index.

After some googling and research without very significant solutions I decided to apply a workaround by avoiding the old API and I rewrite my method.

So, in order to get table indexes information, I have chosen a direct SQL query into SYS tables: sys.tables, sys.indexes, sys.schema.

Because I preferred getting also information about the index’s composed fields, I applied a second additional SQL query:

and I have collected data into a container of defined structure according to my SQL Indexes interest information:

The last member vectColumns stores information about the columns that are used for a specific index.

Finally, the new method that collects table indexes information looks like:

In this way I have complete information about the indexes of my tables.

Conclusion: When the C/C++ API doesn’t give you any hopes don’t forget that SQL saves you.

Share

HTML files generation using XML and XSLT with Microsoft XML DOM API

This short tutorial shows how easy it’s to generate reports in HTML pages using Microsoft XML DOM API together XML and XSLT.

XML (Extensible Markup Language) became a universal standard of encoding data in a format that is both human-readable and machine-readable. It’s widely used in business applications and even Microsoft Office uses it into internal file formats.
XSLT is used for XML documents decoration. Once we have data into a XML files, using the XSLT (Extensible Stylesheet Language Transformations) we can easily generate HTML and xHTML files. XSLT is a W3C recommendation still from 16. November 1999 and in the meantime it was extended with a new version XSLT v2.0.

XSLT uses XPATH to get the XML’s tags information, complete the predefined temples and transform results into a .html document.
Each decent browser has support for XML and XSLT. All we have to do it’s to link two such files (.xml and .xslt) and once we execute the XML file the browser will generate and render our XHTML content.

But in case we are writing non-browser applications the HTML generation becomes a bit complicated in case you are not satisfied with hard-coded solution and want a flexible solution.

Using the Microsoft’s XML Core Services (MSXML) our job became a piece of cake. We focus once over the HTML generator and later in case we want to change something into our look and content we have to deal only with the .xml and .xslt files.

Because of using COM don’t forget proper the calls of CoInitialize() and CoUninitialize().
Here are two samples files generated with the test application using upper method: sample_1, sample_2.

The combination of XML, XSLT and XPATH offers a very flexible way to generate HTML files. With such an approach even native application does not need to change in case we change the HTMLs look. Within the presented case the hard-coded solutions are avoided and most probably a new recompilation is not needed in case we want to change data content (XML) or the look (XSLT).
In case you want to add sophisticated html code (ex. colored, formatted, images, etc.) you need to convert that code into XHTML format before adding data into the .XSLT file.

demo application (932)

Share

Adventures with _chkstk

Called by the compiler when you have more than one page of local variables in your function.
_chkstk Routine is a helper routine for the C compiler. For x86 compilers, _chkstk Routine is called when the local variables exceed 4K bytes; for x64 compilers it is 8K.

That’s all that you get from _chkstk()’s msdn web page. Nothing more…

Overview
A process starts with a fixed stack space. The top of a stack is pointed to by the ESP register (Extended Stack Pointer) and this is a decrementing pointer. Every function calls results in a stack created for the function inside this Process Stack. Every thread function has its own stack. The stack is a downward growing array. When a function starts, the default stack reservation size is 1 MB.
This is contrasting with the heap’s size whether theoretically increases to a limit of 4 GB on 32bits OS. See more information here.

Every thread under Windows gets its own block of contiguous memory, and while function calls are made, the stack pointer is increasing and decreasing. In contrast, a different thread within the same process might get a different block of contiguous memory – its own stack. When a context switch occurs, the current thread’s ESP (along with the IP and other registers) are saved in the thread’s context structure, and restored when the thread is activated the next time.
To specify a different default stack reservation size for all threads and fibers, use the STACKSIZE statement in the module definition (.def) file. To change the initially committed stack space, use the dwStackSize parameter of the CreateThread, CreateRemoteThread, or CreateFiber function.
Most stack problems occur in overflows of existing stacks, as their sizes are fixed and they cannot be expanded.

_chkstk() increases the stack when needed by committing some of the pages previously reserved for the stack. If there is no more physical memory available for committed pages, _chkstk fails. When you enter a function (VC++ with the stack checking enabled), it will call the _chkstk located in CHKSTK.ASM. This function does a stack page probing and causes the necessary pages of memory to be allocated using the guard page scheme, if possible. In this function, it is stated that when it encounters _XCPT_GUARD_PAGE_VIOLATION, the OS will attempt to allocate another guarded page and if it encounters _XCPT_UNABLE_TO_GROW_STACK then it’s a stack overflow error. When _XCPT_UNABLE_TO_GROW_STACK is encountered, the stack is not yet set up properly, that is why, that it will not call the catch because calling it will use invalid stack variables which will again cause another exception.

Case – Too many or too big variables on stack
As I said on top, the function stack size is 1 MB. If you miss that and you’re trying to define and use internally an array like this:

When you’ll compile with VC++ compiler in debug mode you will have a big surprise: the application is crashing on _chkstk() in the moment the _chkstk() tries to create new memory page on stack and fails.
The output window shows next message:
First-chance exception at 0x004116e7 in testApp.exe: 0xC00000FD: Stack overflow.
Unhandled exception at 0x004116e7 in testApp.exe: 0xC00000FD: Stack overflow.

This happens because the 1MB limit is overloaded even on a win32 OS: 4000*200*4 = 3.2MB (approx.).
Same story if you define many local variables and their stack usage overloads the 1MB limit. Off-course the thread stack size can be changed but think once again if it is really needed to do that.
If you really need this big array then the best solution to avoid this crash is using the heap.

Case – Recursive functions
If you have an infinite recursion then you will gate same stack overflow error and the application crashes in _chkstk.asm. Recursive function is not the subject of this article so I don’t go in deep… Here it is a good example of what happens with recursive functions.
The solution is to avoid using recursive functions as much as possible and try to implement an iterative function.

Case – A stack corruption
I have started looking over _chkstk() function in the moment when I got few bugs with crashes with some similarly details. I had to analyze some .dump files and solve few bugs that contained a call stack with _chkstk() on top.
Most of the .dump files call stack contained a second similarly thing: the call of a thread function (so called ThreadFoo()) that was running in a threads pool.
In that moment I started to research why _chkstk() fails and my first track was to debug the stack overflows. I followed a MSDN debugging tutorial and unfortunately I didn’t find something strange. I checked if the local stack variables are not so big in order to fill the ThreadFoo() function’s stack and it did not.
Then a new study of ThreadFoo() function has followed in order to detect the internal functions calls that can fail in some circumstances. I stopped to some trace function calls and I studied deeply. Those trace functions where defined in an external debug class and each time when a new trace file was added it used an internal buffer (TCHAR szBuff[2048] = _T(“”);).
The writing of this buffer was done using: swprintf(). As we know this function is unsafe and is not recommended to use. As long as the content of these trace lines was dynamically build (in some cases those line may contain even dynamically build SQL queries that failed) then the length of these trace lines could be higher than 2048 bytes and then guess what: a stack corruption appears! UPS! The stack pointer will be corrupted (the classic stack overflow case).

So I have implemented and used the next macros:

Now, if we’re using the safe macro we will have no issues.

A safety alternative way to that buffer was the heap using but the heap access is not fast as the stack access so I preferred this approach (in a business application every milliseconds matters for the log system).
After that fixed I met no other stack corruptions in ThreadFoo() and other code areas.

Even if the top of the call stack was _chkstk() this was not the function that failed. The error appeared because of that stack corruption and _chkstk() has just detected.

Conclusion
If your code produces a stack overflow, then you have to rethink your design in right away:

  • If you see _chkstk() on the top of call stack, check if you have no stack corruptions – stack overflow.
  • Don’t try to manipulate the stack by yourself. The default 1MB stack size is basically enough
  • Allocate dynamically if you’re using big arrays
  • If you have recursive functions producing a stack overflow, re-write them using loops (a tip: it is a proven fact that any recursive functions can be programmed non-recursive)

References
Set stack size
Thread Stack Size
_chkstk Routine
Stack (data structure)
Debugging a Stack Overflow – tutorial
Visual C++ apps crashing in _chkstk() under load
Optimization Note (C++) 1: push, pop, call _chkstk
What is Recursion?

Share

pre vs. post increment operator – benchmark

Compiler: Visual C++ 2010
Operating System: Windows 7 32bits
Tested machine CPU: Intel core i3
Download: preVSpost (demo project) (877)

A recent Visual C++ team’s comment on twitter.com reminded me a hot topic that exists in C++ programming world: there is a long discussion of using pre versus post increment operators, specially, for iterators. Even me I was witness to a discussion like this. The discussion started from a FAQ written by me on www.codexpert.ro.

The reason of preferring pre increment operators is simple. For each post-increment operator a temporary object is needed.
Visual C++ STL implementation looks similarly with next code:

But for pre-increment operator implementation this temporary object is not needed anymore.

In the discussion that I mentioned above, somebody came with a dummy application and tried to prove that things have changed because of new compilers optimizations (the code exists in the attached file, too). This sample is too simple and far away to the real code. Normally the real code has more code line codes that eat CPU time even if you’re compiling with /O2 settings (is obviously).
Base on that VC++ team’s tweet related to viva64.com’s research I decided to create my own benchmark base on single and multicore architectures. For those that don’t know Viva64 is a company specialized on Static Code Analysis.
Starting from their project I extended the tested for other STL containers: std::vector, std::list, std::map, and std::unordered_map (VC++ 2010 hash table implementation).
For parallel core tests I used Microsoft’s new technology called Parallel Pattern Library.

1. How the tests were made
1.1. Code stuff
In order to get execution time I used same timer as Viva64 team (with few changes). Each container instance was populated with 100000 elements of same random data. An effective computing function was repeated 10 times. Into this function some template functions are called for 300 times. The single core computing function contains loops like this:

For the parallel core computing the first simple for loop has changed in:

Where cnt is an instance of combinable class and the sum of partial computed elements is obtained by calling combine() method:

As you can see, the parallel_for function uses one of the new C++ standard features: a lambda function. This lambda function and the combinable class implements the so called parallel aggregation pattern and helps you to avoid the multithreaded common share resource issues. The code is executed on independent tasks. The reason that this approach is fast is that there is very little need for synchronization operations. Calculating the per-task local results uses no shared variables, and therefore requires no locks. The combine operation is a separate sequential step and also does not require locks.

1.2. Effective results
The tests were running on a Intel core i3 machine (4 cores) running Windows 7 on 32bits OS. I tested debug and release mode for single and multi cores computation. The test application was build in VC++ 2010 one of the first C++11 compliant.
The OX axis represents the execution repeated times, and the OY axis means time in seconds.

1.2.1. Single core computation
Debug

Release

1.2.2. Multi cores computation
As you know, multi core programming is the future. For C++ programmers Microsoft propose a very interesting library called Parallel Pattern Library.
The overall goal is to decompose the problem into independent tasks that do not share data, while providing a sufficient number of tasks to occupy the number of cores available.

This is how it looks my task manager when the demo application runs in parallel mode.

Isn’t it nice comparing to a single core use? 🙂

Debug

Release

1.2.3. Speedup
Speedup is an efficiency performance metric for a parallel algorithm comparing to a serial algorithm.

Debug

Release

Conclusions:
The biggest differences appear in the debugging area where the pre-increment is “the champion”.
With primitive types (like int and pointers), the opposite might be true, because of the pipe-lining that a CPU does. With post-increment, due to optimizations in release there is no copy to be returned for these simple types.
According to these results I have to agree with Viva64 team. Even if the results are so close in release version I keep my opinion that using pre increment operator is preferred instead of post increment operators. We all know how long it takes the debug period and how important is every second that we win in long debugging days.
If you still have doubts in using pre-increment operator or you need a flexible way of switching this operators in your code you can easily implement some macros like these:

#define VECTOR_ITERATOR(type, var_iter) std::vector::iterator var_iter;
#define VECTOR_FOR(vect, var_iter) for (var_iter = vect.begin(); var_iter != vect.end(); ++var_iter)

Share

Numeric type conversion to std::string and vice versa

In our real applications we have to convert from strings to integer or to real variables and vice versa (double/float/int variable to std::string).
We can realize these conversions using C style CRT function or we can try C++ approach via STL.
Unfortunately, current C standard libraries do not offer a complete support for any type of conversion. For instance, if we try to put an integer into a C++ string object (std::(w)string) using a well known function itoa() then we get next error:

// error C2664: ‘itoa’ : cannot convert parameter 2 from ‘std::string’ to ‘char *’

A C style approach in order to avoid this error means using an intermediary buffer:

Same story if we try to convert a std::string to an int:

// error C2664: ‘atoi’ : cannot convert parameter 1 from ‘std::string’ to ‘const char *’

In this case we can use c_str() in order to return a constant pointer to char.

An elegant way to get rid of such problems is to build two conversion function that use templates and C++ streams.
Base on this idea, I created a Sting2Numeric class that contains two static methods: Type2String() and String2Type().

where BadConvertion is a std::runtime_error‘s derived class.

Because of ANSI and UNICODE project’s compatibility I defined few macros:

Because of this compatibility I strongly recommend using a xstring alias instead of std::wstring or std::string.
When you want to convert an int, float, double, or other numerical type to a xstring in a C++ style you can use the Type2String() function. Vice versa, if you want to convert a xstring to these types you can use String2Type().

In order to avoid possible thrown exception I recommend to you using a try catch block whenever you’re using these functions. I prefer using xstring for string/wstring variables definition, too.
Here is a sample of using this class:

The String2Numeric class can be extended. For instance, if the conversion throw an error then you can add detailed information in the exception message.

Download String2Numeric (840) class.

Share

File size fast detection

Many times in our job, we need to work with files and need to know file properties.
One of the most important properties is file size. Of course, there are a lot of API that allows finding this property, but most of them needs additional file operations: open file, find file size and close file.

A direct and fast way in order to detect the file size without these operations means the CRT run-time library’s function _tstat64() and stuff.

In the header file or on .cpp file please add next macros definitions:

Then, write next function:

If you’re using WinAPI there is an even faster way in order to get file size.

Finally, call these functions wherever you need.

Share

Dynamic popup and drop down menus for custom representations

Many applications allow dynamic customization for visual objects or data views. For instance, well known Internet Explorer application provides toolbars customization using a popup menu that appears when the user execute right click mouse action in toolbar zone area.

Internet Explorer sample menu

Other sample where this kind of menu is very useful is when it’s used in order to customize database data representation in Windows controls like List control or grid control. These kind of applications allow data filtering and show/hide columns using this kind of menu. The user just right click on control header and gets what he need.

Starting from this idea, I implemented a class CDynamicPopupMenu. This class allows an easy building of this kind of menus. I used if in a demo dialog base application over a list control.

my demo application

Internally, this class uses a STL container (std::map) with a data structure used in order to embed items menu properties. When the menu is built, menu’s behavior is implemented using these properties.

Add new menu item
The new item add menu method has next definition:

where:

  • item_id – represents internal item ID; the ID is used for menu customization, too;
  • parent_id – parent item ID used when we define a new items sub-group (a drop-down menu); the attribute value is 0 if menu item is a part of initial menu;
  • is_visible – this flag is used a item is checked / unchecked. In my demo application this flag is set true for all list control’s columns that we want to display. For “Select All” and “Check All” items this flag is false because we want to create new subgroup that contains new columns, but we don’t have “Select All” or “Check All” columns.
  • check_flag – this flag allow check/uncheck menu property;
  • has_child – if is true allows a subgroup definition (a new drop-down menu);
  • item_name – unicode menu item name;
  • enable_flag – defines if the item is enable or disable.
  • Add separator item
    Add separator item method definition looks like this:

    where:

  • item_id – menu item ID;
  • parent_id – parent item ID from the subgroup has started; the attribute value is 0 if menu item is a part of initial menu.
  • Menu add items sample
    In my demo application, in CtestPopupMenuDlg::SetDefaultMapValues(void) method, among other things, you can find next calls:

    Get menu internal data
    In order to access the internal data container (std::map) that stores all dynamic menu items you just can use next method:

    followed by:

    Create and display menu
    Menu creation must be done just after we add all menu items. The menu is displayed only after TrackPopupCustomMenu() call. The definition of this method looks like this:

    where:

  • point – mouse coordinates where the menu start building;
  • hWnd – parrent window handle where the menu is created.
  • Function’s return value is the menu IDs that was selected. If no item was selected the function returns 0.
    In my demo application, menus creation is called on list control right-click method (NM_RCLICK).

    As you can see, I’m calling TrackPopupCostumMenu(), using mouse point property when the user right-click over list control.
    I am saving list control handler, selected item ID and WM_NOTIFY value into a pointer to message notification structure NMHDR. Then I’m passing this pointer to OnNotify() method.
    Using WM_NOTIFY message and OnNotify() method, I inform parent control window that a new event was generated.

    I am calling GetItemCheckedFlag() if order to detect selected item check status (check / uncheck). Then, if item state means check I apply negation over this bool flag and I’m calling SetCheckedItemFlag() method. Finnaly this method produce changes in my control list, depending on my menu command (FillData() method).

    Menu interaction with parent window (list control)
    In my demo application, the interaction between dynamic menu and list control to be treated by FillData() method.
    In order to use CDynamicPopupMenu’s internal container data is need to initialize a DynamicMenuData pointer with GetDynamicMenuData()’s returned value.

    Using that pointer to internal menu data, I iterate over internal container, and for those items that are visible and selected set on true I insert columns in my list control.
    Similarly, when using such menus, the application can apply filters on real data.
    CDynamicPopupMenu class contains other useful methods. This kind of menu can be used in different situations in order to change application’s behavior.

    Download demo application: testPopupMenu (Visual C++ 2005 project)

    Share