Thanks to everyone who participated in today’s working group call! We had a great discussion regarding specifics of the QA Specification, which is summarized in the call notes below. A recording of the call will be posted soon on our Technical Forum section of the ICP website.

Please, continue to review the QA Specification on GoogleDocs and provide us with your thoughts and comments. We are hoping to have a complete draft of the document prepared for the next call in late July, at which point we will discuss any final thoughts or comments, with the idea of releasing the document in mid to late August. - ICP Team

• Energy data - ensure that we address filling data gaps (UT3 for example)
  
  
• End use energy usage - suggest using ASHRAE Level II End Use Analysis workbook, to build up the end use energy usage from collected building information (lighting, nameplate data, etc)
  
  
• Weather - mention using weather not just for baseline development, but also for calibration of the energy model, energy savings calculations, and M&V efforts
  
  
• References - we don’t want to completely recreate guidelines and protocols that are already developed; but need to ensure we include enough critical information from these documents, particularly for documents that are not free, so the reader understands what is required. The resources can be referenced to provide greater detail, but the critical components should be included in the QA Specification
  
  
• Retrofit isolation baseline - make sure it only applies to the system within the defined boundary. Applies to Option A and B approaches, and projects such as lighting retrofits, boiler retrofits, PV, etc. Refer to the CCC Guidelines for Verifying Savings from Commissioning Existing Buildings
  
  
• Load profiles - refer to ECAM for guidance with this section
  
  
• Baseline energy model development - should this be moved to the baseline section, or leave it in the savings calculation section? Is it considered part of the baseline development, or is it a “tool” used as part of the Savings Calculation?
  
  
• ECM description - consider mentioning that a “persistence plan” should be included (this is already covered in the OM&M section however)
  
  
• Financing / Investment Criteria - EUL is already included in this section (along with many other components)
  
  
• Mention that the financial metrics should consider uncertainty if possible, as well as persistence of energy savings
  
  
• Cost estimates - not a lot of great resources; RS Means, but what else? Mention that project should utilize third party contractor estimates whenever possible. Particularly if the building owner already has a relationship with specific contractors.
  
  
• ECM modeling - discuss keywords used, assumptions, and documentation of sources. As part of QA, cross reference assumptions with construction documents
  
  
• Operational Performance Verification - group still agrees it is appropriate to not use the term commissioning (Cx); but can still reference / use Cx resources to develop this section. Utilize construction checklists as well
  
  
• Systems Manual - will reference ASHRAE’s new Process and Procedures for Development of a Systems Manual (the document should be approved next week)
  
  
• Did not get to the OM&M and M&V sections of the QA Spec. Will discuss specific sections of these two in the Technical Forum, to get input from the Working Group.

The ICP Quality Assurance Protocol Technical Forum is making good progress on the development of the QA Protocol.  

Our next meeting will occur on Tuesday 6/24 at 10am PDT / 1pm EDT.  If you are interested in attending and are not on our QA Technical Forum mail list please send us a note for the call-in and meeting information.  

The  latest version of the tool can be viewed and commented on here: https://docs.google.com/a/sevengenergy.com/document/d/15rRJWoRTWx80Z8L12PqteCeXsz7awzmDuA7l854msBQ/edit#

And the current agenda for the upcoming meeting can be accessed here: https://docs.google.com/a/sevengenergy.com/document/d/1lqcTLrT6iAp513JxcBMsx1YsjbnmNJIMe5geGaor7E0/edit#

Thanks to all for their participation.

The ICP Team

The QA Specification is going to address uncertainty and risk. While we do not want to provide an exhaustive description of this topic, we do want to address it in a clear, concise way. The following is an excerpt from the QA Specification addressing this topic. We are asking the QA Technical Group, and the community at large, to comment and provide suggestions for this section.

The following is the current form of this Uncertainty and Risk section. Please send your comments or suggestions to [email protected]. Thank you for your continued support with this effort!

Uncertainty and Risk
The estimated energy-cost savings associated with the ECMs and package of measures is a critical value for investors considering EE projects. Unfortunately, savings estimates are typically calculated as a single number and do not indicate a probable range or an estimated uncertainty. Failure to provide information about uncertainty leaves the financial analyst with no means to price the appropriate rate of return. This causes the financial analyst to increase the required rate of return or to de-rate the savings before applying the financial model. This practice undermines the viability of energy projects (Mills et al. 2003).

For most uncertainty analysis methods, the inputs (assumptions) need to be specified as ranges, and their distribution type specified (such as normal, lognormal, uniform, log-uniform, etc.).  A statistical sampling method is then applied to develop sets of parameter values for the assumptions, representative of all possible combinations. The calculations are then performed with these sample sets, and a probability distribution function can be developed and reported, indicating the uncertainty.

For projects utilizing spreadsheet calculation methods, automated calculation tools can be employed, such as the Monte Carlo simulation method.  During a Monte Carlo simulation, values are sampled at random from the input probability distributions.  Each set of samples is called an iteration, and the resulting outcome from that sample is recorded.  Monte Carlo simulation does this hundreds or thousands of times, and results in a probability distribution of possible outcomes. Various applications are commercially available that can be applied as an add-in to Excel, and used to automate this Monte Carlo simulation approach.

For projects utilizing an energy model, a similar approach can be applied. Values are sampled from the developed input probability distributions, and then multiple iterations of the energy model need to be performed. While automated or standardized approaches to this uncertainty analysis have yet to penetrate the energy modeling venue, work is currently underway to develop options for completing uncertainty analysis. In particular, the next release of OpenStudio, an open-source application suite and software development kit which supports whole-building modeling using EnergyPlus and daylight analysis using Radiance, will feature uncertainty analysis tools and methods that draws from the DAKOTA project, an engineering optimization and uncertainty analysis modeling library developed by Sandia National Laboratories. This functionality will allow the assessment of sensitivity of the output to the input parameters, those parameters that contribute the most variance, and how they interact with each other. 

In many cases, resources and time may not be available to determine the uncertainty associated with a project. A cost- effective alternative to quantifying uncertainty is to reduce risk. This is accomplished by:

• Reducing the number of assumptions used in the savings calculation and cost estimation efforts.
  
• Utilizing conservative assumptions when these inputs are necessary.
  
• Applying best practices to all components of EE project development.
  
• Properly applying design, delivery, and operational processes.
  
• Training facility staff adequately.
  
• Performing operational performance verification.
  
• Providing systems and methods to monitor and track performance on an ongoing basis, and providing an adequate managerial and recognition / response plan.
  
• Performing a comprehensive quality assurance process on all components of the EE project development, avoiding bias at all costs.
  
  

These practices will reduce risk, providing the necessary level of confidence to an investor-ready EE project.

A particular challenge to energy efficiency project development, especially some multifamily projects, involves access to utility data. Critical to the development of a project's baseline, these data can be difficult to obtain, especially considering tenant privacy laws. Even when tenant privacy is not an issue, access to these data in a usable format can be challenging.

Consider this example: I am currently a member of my hometown's energy and sustainability committee. As part of our efforts to revise our city's master plan, we have been working with our utility to obtain aggregate monthly electricity and natural gas consumption data for the residential and commercial/industrial sectors within our community. Not individual customer's usage data (which would butt heads with tenant privacy laws), but aggregate data for the entire city. We started working with the utility to obtain these data six months ago. And we're still waiting...

But the winds of change are blowing, and gaining speed. Presented for consideration are two examples of significant developments on the data access front.

Green Button
Green Button is an industry-led effort determined to provide electricity customers with easy access to their energy usage data in a consistent and easy-to-use format. This data access concept is based on a common technical standard developed in collaboration with a public-private partnership supported by the Commerce Department's National Institute of Standards and Technology. 

The idea behind Green Button is to provide a secure, consistent, and easy-to understand method for downloading data directly from the utility's website. At current count, there are 31 utilities and energy suppliers that have implemented Green Button, and nine additional utilities have expressed a commitment to implement Green Button.

Providing access to utility data in a consistent and easy manner not only provides these data to end-users, but also provides a platform for developers to create innovative web-based, smartphone, or other platform applications. These tools can deliver a host of potential services, things like rate plan selection assistance, renewable energy sizing and financing, virtual energy audits, measurement of energy efficiency investments, or energy usage visualization.

E-Access
Another exciting development in the utility data accessibility world recently came from Senators Ed Markey (D-Mass) and Mark Udall (D-Colo) who joined together to introduce the Access to Consumer Energy Information (E-Access) Act on March 27, 2014.

Like Green Button, this bill seeks to improve access to energy consumption data, providing incentives to utilities and states that offer utility information to commercial and residential customers. If the bill is enacted, state activities meant to improve customer access to and understanding of energy usage would qualify for government funding. 

Furthermore, this E-Access Act would require the DOE to create guidelines for improving access to electric energy information, and would address specific issues including time frame, consumer data, privacy protections, security, and how to present the information in a readily understandable format.

Building Momentum
While access to utility data is still challenging, the market is clearly building momentum to improve this situation and overcome this long-standing hurdle to energy efficiency project development. Green Button, complimented by the E-Access Bill, represent two powerful tools that will help continue to push the industry forward, improving data accessibility, and promoting innovative uses for data that were, until recently, difficult to access or receive in a consistent, usable, easy-to-understand format.

Tuesday, April 29, 2014
1 pm EST (60 min)Conference Number: (415) 854-0868

Compile and discuss available resources, tools, and best practices regarding quality assurance methods.

QA Checklist Elements:

• Data - historical energy usage
  
• completeness of data collection and validity of data
• utility data analysis - stability and normalization
  
• retrofit isolation baseline development

• Data - and building system operation / building asset data
  
• building data collection methods and processes
• asset data collection
  
• operational and performance data collection
• Data - load profile
  
• data collection and analysis
  
• Investment criteria
  
• SPB, SIR, IRR, NPV
  
• Cost estimates
  
• Implementation cost reasonableness
  
• Energy modeling
  
• quality assurance and calibration
  
• ECM modeling
  
• Spreadsheet calculations
  
• quality assurance regarding methods and assumptions
  
• reasonability review methods
  
• Operational performance verification
  
• resources
  
• report templates
  
• system manuals - concise
  
• Operations, maintenance and monitoring
  
• FDD tools / resources
  
• RCx tools / resources
  
• Operator’s manual templates
  
• Training
  
• M&V
  
• Automated tools
  
• Resources
  
• Templates (M&V plans and reports)
  
• Uncertainty and risk
  
• methods for determining / quantifying uncertainty
  
• methods for determining risk
  
• Roles of third-party evaluators
  
• currently suggest third party performance of Cx
  
• currently suggest third part performance of M&V; at least require third party review of M&V
  
• Other QA topics that should be addressed?

The Investor Confidence Project (ICP) is pleased to announce our next Technical Forum call, which will focus on the development of the Quality Assurance (QA) Specification.  This significant undertaking will address areas of an energy efficiency project that require greater levels of scrutiny and QA, as well as methods, roles and responsibilities associated with the QA process.  The QA Specification will address QA processes and resources applied to each element-procedure-documentation item in the three protocols (Large, Standard, and Targeted Commercial) in a standardized and repeatable process. The ultimate intent is to provide a framework for best practices, and links to QA resources, to assess and minimize risk and uncertainty associated with energy efficiency projects and the cost and complexity for investors to gain confidence in projected savings and therefore cash flows that result from investment in energy efficiency.

The QA Specification will utilize existing guidelines, processes and methods already established in most cases.  If resources for a particular item are not available, a QA method specific to this item will be developed.

This first Technical Forum call is scheduled for Tuesday, April 29th at 1 pm EST.  This call will provide a forum to discuss various industry-accepted methods associated with the protocol tasks, as well as brainstorm available tools and resources for QA.  A complete agenda for this call will be forthcoming next week.

We look forward to a lively discussion with all participants.

Regards,

The ICP Development Team

Multifamily Technical ForumAgenda1 PM EST, Tuesday, March 25, 2014

1. Discuss final version of Large Multifamily Protocol. Changes from previous draft included:
   
   
1. Baseline: mentioned specifically excluding data period covering major renovations from baseline development.
   
   
2. Baseline: added “physical attributes and equipment inventories” requirements specifically to Section 2.1. Added additional text to Section 4.2 requiring detailed descriptions of ECMs.
   
   
3. Savings Calcs: Model calibration - changed suggested accuracy to 10-15%. with CV[RSME] 15%.
   
   
4. Savings Calcs: Life cycle metrics - kept simple payback as an option. Eliminated recommended SIR>1 (from all protocols).
   
   
5. Savings Calcs: Added a line in section 7.2 mentioning that uncertainty does not need to be quantified, but activities should be employed to minimize uncertainty and risk.
   
   
6. M&V: Added note in M&V section regarding vacancy rate as an independent variable.
   
   
2. Discuss Standard version of Multifamily Protocol
   
   
1. Baseline: retrofit isolation baseline
   
   
2. Savings Calculations: using non-energy modeling savings calculation methods
   
   
3. M&V: options A and B
   
   
3. Discuss Targeted version of Multifamily Protocol

Attending Members: 

• John Jones; Maggie Selig; Ian Shapiro; Chris Diamond, Darrien Crimmin, Tracy Phillips, Matt Golden, Jeff Milum
  

1. Technical Forum section of the ICP website - portal for communication
2. Specific details regarding the actual energy audit and process are not covered in the protocol.  But should we provide more direction regarding building asset data to be collected (Section 2.1 - mention physical attributes, etc.), and more detailed scopes of work for ECMs (is this addressed adequately in Section 4.3?)? Mention equipment inventory in particular; better defined scopes of work for the ECMs.
3. Model calibration - to avoid overestimating savings, require that calibration be performed to 0 to -10% or -15%. Eliminate +15% piece. CV(RSME) – just 15%, not plus or minus, on a monthly basis. FEMP says +/-5% on annual basis. If we say only -15%, that’s fine.  But don’t allow this to give a “false sense of security.” Still need to check reasonableness of savings calculations. QA mention should be put into the baseline section too, similar o what is included in the ECM (Savings Calculation) section.
4. Life cycle metrics - make this a requirement instead of optional? Eliminate simple payback as an acceptable metric? Keep simple payback as an option. SIR as a requirement.
5. Large / standard / targeted project sizes - can number of units be used as a metric for project size? (Eg., ~$3k in potential ECMs per unit, so 150+ units = large, 50-150 units = standard…). Other resources suggest large = 500+ units.  Or is this based on project cost like commercial (large = ? $1M) I don’t think this matters.  I think we keep the definition as we have it in the large and standard commercial protocols (in general: >$1M = large; $200k-$1M = standard).
6. How do we deal with parking in the square footage - do we need to address parking in the protocol? Attached parking garages are included in the square footage – we will consider mentioning that this is the case so that it is clear in the protocol.
7. Baseline development

• We mention noting major renovations; should we include language that specifically excludes these utility data from the baseline development? Yes, include this language, and make sure the protocol indicates that the data since the renovation was completed should be used to develop the baseline – not data during or before the major renovation.
  
• Is there a “standard” sampling approach we can suggest for tenant billing? NYSERDA MPP v5 and NYC LL84 requires that 10% of tenant billing (for each “apartment line”) be included to estimate energy usage for the building’s “tenant” portion; is there a HUD resource? Still no leads on the HUD resource. Will keep looking for this. This NYSERDA guidance on a “sampling” approach is the only one I know of so far.
  
• Use of “default” values for tenant energy usage? NYC LL84 allows use of default values (provides these) for NY properties. Can this approach be used?  Are there other sources of these data for other regions?  This is the only regional source I know of.  Could put in the protocol that this approach could be used if a credible data source is available.
  
• Vacancy rates or other related adjustments are primarily a concern when they are anticipated to be different between the pre and post construction periods - should handling these be specifically mentioned in baseline development and M&V activities (can affect tenant energy usage and central heating plant energy usage) This is really an independent variable. Needs to be properly defined in the baseline, and adjusted for in the M&V during the regression analysis.
  

8.      Savings calculations and implementation cost estimates - discuss uncertainty in more detail? Never done on actual EE projects. Maybe more discussion of it in the M&V section though?

    9.    SIR - should we define this specifically in the protocol? We require an SIR>1 Maybe a definitions section, or a separate document with definitions to start with? Check that all of the acronyms are spelled out the first time, and maybe include links to definitions of these as links in the protocol.
We should keep in mind that there are other items included in a project, or specific items that don’t pay back very well (like a boiler replacement, window replacement, other non-energy measures). So SIR>1 should not be a requirement.

   10.  Affordable housing - good part of the utility costs are picked up by the government, so you have a third player. There are many different types of lenders, and they compete regarding whose mortgage takes priority.  Finance side is complicated.  The MF protocol should apply, if a project development team can work out the financial side of an affordable housing scenario. So no need for a different affordable housing protocol? No need for a separate protocol for affordable housing.

   11.    Sections 5, 6 and 7: are these too long?  Are they specific enough?

Should we include an energy audit section in future iterations of the protocol?  We should consider this.  At the very least, refer to the NYSERDA modeling guidelines or an equal guideline.