Learn how VERISCO's test system checks if the ISO 15118 implementation is interoperable against other implementation on the market.

‍

‍

What do you do if you are working on an ISO 15118 implementation, be it for an electric vehicle (EV) or a charging station, and want to make sure that it is a) interoperable with other implementations in the market and b) conform to the standard itself. Currently, you only have two options: You attend the International CCS & ISO 15118 Testing Symposium, or directly meet with a supplier of a test system, such as VERISCO. I did both, aiming at making RISE V2G (GitHub) a bulletproof reference implementation of this standard and thus your go-to solution to test your product, or even use it in your products. RISE V2G is free and licensed under the MIT license.

The path of RISE V2G

I have put a lot of effort into RISE V2G over the past couple of years. It started out as a university project back at my days at the Karlsruhe Institute of Technology (KIT) where I was researching on how to integrate electric vehicles with bi-directional power flow into the smart grid through information and communication technology. 

Eventually, it matured over time and became an open source project on GitHub in June 2015. I have attended all ISO 15118 Testing Symposia except for one in 2015 that took place in Tokyo. Back then, I worked at a startup whose primary focus was not on ISO 15118. 
Well, since March 2016, I am a freelancing consultant and created my own brand V2G Clarity. Ever since, I am operating under this brand to drive the advancement of ISO 15118 itself and spread the knowledge about this promising new charging and communication standard for electric vehicles.

I genuinely believe that ISO 15118 will revolutionise the way we experience charging an electric vehicle, be it a car, a bus, or even a ship or e-bike. Second, trust in a new technology comes with a worldwide adoption as well as open access to it that facilitates interoperable solutions. After all, interoperability is one of the key enablers for successful market adoption. 
‍
This is exactly why I put so much effort into RISE V2G to make it THE open source reference implementation of ISO 15118. And my meetings with Verisco brought me one significant step closer to this goal. With that being said, let me introduce you to VERISCO.

VERISCO - testing services for V2G communication

VERISCO, an acronym for Verified Smart Communications, has developed a highly modular and automated test equipment for the Combined Charging System (CCS) of Electric Vehicles and Charging Stations. They provide conformance and interoperability testing services according to ISO 15118, DIN 70121, IEC 61851-1 and help you to reach cross-vendor interoperability.

It is a spin-off from the Communication Networks Institute (CNI) of TU Dortmund in Germany. Its team, as shown in the figure below, consists of interdisciplinary engineers who continuously contribute to international standardisation communities. In fact, the founders of VERISCO, Jens Schmutzler and Sven Gröning, are both in charge of defining the ISO 15118-4 and -5 documents that specify the conformance tests for ISO 15118-2 and -3 respectively. Both have also been responsible for incubating the International CCS & ISO 15118 Testing Symposium and have been organising these events ever since.

‍

‍

Have a look at their products & services page to see how they can help you to ensure reliable and secure solutions enabling user acceptance and economies of scale for your products.

How does the testing procedure work

I met with VERISCO twice over the last couple of weeks for test sessions which lasted four days in total. The first time I visited, I expected to be done after one day of testing. After all, RISE V2G was already a very mature implementation which covers all features the ISO 15118 standard has to offer.

Boy, was I wrong. Don’t underestimate the vast number of good and error test cases they have implemented in their test system to uncover every possible faulty implementation detail.

Well, the ISO 15118-2 standard alone lists more than 800 technical requirements that need to be tested. It’s insane, I know. Yet, you can group some of the requirements into different feature or message sets. Let’s start with the groups of features you can test:

• The ISO/OSI communication layers 1 (Physical layer) and 2 (Data link layer) addressed by ISO 15118-3, including the Signal Level Attenuation Characterisation (SLAC) mechanism. SLAC is used to establish a communication link between the EV and the charging station.
• The ISO/OSI communication layers 3 (Network layer) up to 8 (Application layer) addressed by ISO 15118-2
• ISO 15118-2 message sets related to External Identification Means (EIM, one of the two ways of authenticating and authorising an EV for charging)
• ISO 15118-2  message sets related to Plug & Charge (PnC, the other way of authenticating and authorising an EV for charging). PnC includes secured communication via Transport Layer Security (TLS), the creation and verification of digital signatures and the exchange of digital certificates (so-called X.509 certificates).
• ISO 15118-2 message sets related to Alternating Current (AC) charging
• ISO 15118-2 message sets related to Direct Current (DC) charging
• Timing requirements for all the test cases mentioned above

As RISE V2G does not feature the SLAC mechanism (yet, maybe this will be added some day), we skipped the physical and data link layer tests and went right to the high-level communication that starts with a so-called SECC Discovery Protocol (SDP) request.
For each test session, you need to agree on which party, either VERISCO’s test system or your system under test (SUT), acts as the EV and which one acts as the charging station. For a complete conformance test also covering optional ISO 15118 features, a complete catalog of settings - and in some cases, even their permutations - need to be coordinated between you and the expert operating the VERISCO test system. After this has been sorted out, the above-mentioned test cases can be executed in a fully automated fashion.

It starts out with a so-called “good case test campaign”, which means that VERISCO’s test system tries to successfully run through a communication session with a short charging loop in a series of test cases. This is to ensure that the messages in your SUT have been implemented correctly. This could be compared to an interoperability test run between an EV and a charging station, similar to what is mostly performed at the Testing Symposia. However, that's the smallest test part.

Next up in the conformance test are the vast number of error test cases in the "error test case campaign". During this campaign, VERISCO’s test system will send messages that should invoke a particular error handling in your SUT.

Example: 
Let's assume that VERISCO's test system acts as the EV and your SUT acts as the charging station. One of the error test cases involves sending a contract certificate (with which the EV authorises itself for charging) within the PaymentDetailsReq message (Req is short for Request) whose validity period already expired. Thus, your SUT is expected to respond with a PaymentDetailsRes message that has its ResponseCode parameter set to "FAILED_CertificateExpired". If your SUT sends any other response code, this specific test case is considered to have failed.

The short video shown below gives you an impression of how such a test run is visualised. What you see here are the individual error test cases on the top left (e.g. TC_SECC_CMN_VTB_CertificateInstallation_002), the logging information printed for each test run on the top right, some test case configuration data on the bottom left, and a visual representation of the messages sent and received in the bottom right. The VERISCO test system also provides other supporting views and logging facilities for effective fault isolation and error reporting.

‍

Results from the test session with VERISCO

RISE V2G has been thoroughly tested on both sides, acting as a charging station and as an EV. And I can proudly confirm that RISE V2G successfully passed all timing and error test cases while acting as the charging station.

I did not have enough time to run through all error test cases with RISE V2G acting as the EV. So I cannot provide the same level of certainty for the EV side. However, it looked very promising so far and I will probably set up another meeting with VERISCO to conduct the remaining error test cases.

Update 15.9.17:
I finally managed to conduct another test of RISE V2G against VERISCO, this time specifically focussing on the EV part. Happy to announce that both sides, the SECC (charging station communication controller) and EVCC (EV communication controller) passed all good and error test campaigns of VERISCO.

What does that mean for you?
It means that you can use RISE V2G as a testing counterpart for your ISO 15118 implementation and, in case the communication works fine, significantly raise your confidence that your solution will conform to the standard and be interoperable to other solutions in the market.

However, don't forget one important fact: RISE V2G does not make the need obsolete to check your ISO 15118 implementation against a test system such as VERISCO's.  It only tells you that there is an extremely high probability that your system works just fine if you do not encounter any implementation errors while testing against RISE V2G. In doing so, it will save you a lot of time (and money) you need to invest in a testing session with a test system provider such as VERISCO.
But RISE V2G does not provide error test cases to invoke a specific error handling in your implementation.

I cordially invite you to download RISE V2G and test it against your ISO 15118 solution. Or to simply play around with it and make yourself familiar with how ISO 15118 works. For a detailed guide into ISO 15118 itself, both for beginners and experts, have a look at the ISO 15118 Manual.

Failed test cases do not necessarily indicate an error in your implementation (SUT)

In some rare cases, VERISCO's test system will report an error test case as failed, although it turns out that it didn't fail. This might sound strange, but let me shortly explain. Some requirements define different negative response codes to be sent depending on the false data being sent by the EV.

Example:
Imagine that the test system, acting as the EV, sends a message that needs to be digitally signed. 
Let's assume that the message is sent in such a way that intentionally both the digital signature - placed in the message's header - and a particular parameter of the message body will be set to invalid values.
We are now in a situation where there are simply two different negative response codes that could be sent, one regarding the signature value (FAILED_SignatureInvalid) and one regarding the specific parameter.
The decision which negative response code will be sent ​​depends on which part of the message is checked first by the charging station.

The ISO 15118 standard does not define which specific negative response code shall be sent in case a variety of negative response codes could be sent. This is probably something that should be fixed in edition 2 of ISO 15118-2.
However, due to the strict requirement-based paradigm of the conformance test specification, the test system will expect only one of those negative response codes for each of the two corresponding test cases. Consequently, it marks the other related error test case as failed if the same negative response code is sent by the SUT - although the behaviour can be considered as compliant according to the ambiguity in the standard.
This way of implementation in the test system actually helps to isolate the exact behaviour of the SUT and to detect potential errors.

The bottom line is, you need to evaluate if a failed error test case is really to be considered as failed. But this is exactly why VERISCO experts are there to help with the interpretation and assessment of testing results as part of their conformance testing services. 

Providers of ISO 15118 interoperability and conformance test systems

VERISCO is not the only provider of an interoperability and conformance testing system. However, they are the first in the market and maybe even provide the most detailed error test cases as they directly contribute their work to the standardisation community by defining the conformance specification of ISO 15118-4 and -5.

To my knowledge, there are three more players in the market which provide test systems. I did not have the possibility and time yet to test RISE V2G against their system, so I cannot tell you any more details. 

These are the other test system providers:

• Vector
• Keysight Technologies
• comemso

‍