Watershed Analysis Monitoring Pilot Project Evaluation
TFW Ambient Monitoring Program
Dave Schuett-Hames & Allen Pleus
Northwest Indian Fisheries Commission
Watershed Analysis is a scientific and regulatory process instituted by the Washington State Forest Practices Board in 1992 to address cumulative effects from forest practices on fish habitat and stream channels. The scientific assessment component identifies areas of resource sensitivity on forest lands where forest practices could alter the rate of sediment, peak flows, large woody debris (LWD) or temperature inputs to stream channels. Linkages (causal mechanisms) between the areas of resource sensitivity and specific channel response reaches where adverse changes to stream habitat could occur are also identified. The regulatory component of Watershed Analysis requires development of specific practices (prescriptions) for each area of resource sensitivity that are designed to prevent or minimize cumulative effects.
The Watershed Analysis rule encourages voluntary monitoring to help evaluate the effectiveness of Watershed Analysis. Monitoring results can be incorporated into the formal review and evaluation of each Watershed Analysis that occurs at five year intervals. In 1994 a voluntary cooperative monitoring module for Watershed Analysis was developed by the TFW Cooperative Monitoring, Evaluation and Research Committee (CMER) and the TFW Administrative Committee. The monitoring module was adopted by the Forest Practices Board and included in version 3.0 of the Watershed Analysis manual.
From October 1994 through September 1995, the TFW Ambient Monitoring Program at the Northwest Indian Fisheries Commission (NWIFC) evaluated the proposed monitoring module on a pilot basis. The project was done with funding from the Washington Forest Protection Association. The purpose of this report is to present the results of the pilot test, discuss opportunities and obstacles encountered by the teams, and identify recommendations for refining the monitoring module.
Pilot Monitoring Evaluation Procedure
The procedures for conducting the pilot test of the Watershed Analysis monitoring module were as follows:
Notification of Watershed Analysis teams that were candidates to develop monitoring plans.
Watershed Administrative Units (WAUs) where Watershed Analysis was underway or completed were identified from the Watershed Analysis tracking log compiled by the Department of Natural Resources (DNR) Forest Practices Division. A letter was sent to the team leader of each WAU informing them that the draft Watershed Analysis monitoring module was ready for use on a pilot basis, the TFW Ambient Monitoring Program was available to provide assistance to teams interested in preparing monitoring plans, and that the experiences of teams using the module during the pilot period would be used to evaluate and improve the module. In addition, each DNR regional forest practice coordinator was sent a copy of this letter and asked to notify project leaders of all the active or completed Watershed Analyses their region.
Initial contact with potentially interested Watershed Analysis (WA) teams.
When inquiries were received from interested WA teams, we provided more detailed information on the monitoring module to the contact person over the phone or during a informational meeting with interested participants in the Watershed Analysis. This typically included answering questions on the current status of the monitoring module and the purpose and advantages of monitoring. Following this discussion, the teams or team leaders typically made a decision on whether to proceed with development of a monitoring plan.
Assisting teams develop monitoring objectives.
Most of the teams that proceeded to develop a monitoring plan were interested in some form of assistance from the TFW Ambient Monitoring Program. The type of assistance we provided included training (explaining the procedure in the manual), facilitating a group process to identify monitoring objectives, and providing support and encouragement throughout the process.
Follow-up during monitoring plan development.
After identification of monitoring objectives was completed, one or more individuals typically tackled the task of compiling supporting information on the monitoring objectives into a monitoring plan report and preparing detailed sampling plans. We periodically contacted these people during this stage to discuss their progress.
Results and Discussion
The observations and insights we gained from working with teams using the monitoring module are presented for each of five steps in the process. The steps are 1) making the initial decision to initiate a monitoring plan, 2) identifying monitoring goals and objectives, 3) preparing a monitoring plan report, 4) development of detailed sampling plans for each objective, and 5) obtaining commitments to implement monitoring plans. Table 1 shows the WAUS which were identified as potential candidates for development of monitoring plans, and the progress of the various teams through these steps.
Step 1: Decision to voluntarily initiate a monitoring plan
The first, and most critical step in voluntary implementation of the monitoring module is a decision by the WA project leader and the participants to proceed with development of a monitoring plan. The majority of the WAUS that were identified as potential candidates to develop monitoring plans never got past this first step. Some of the candidate WAUS did not respond to the information we sent. Others responded to our initial inquiry but have not developed monitoring plans. The team leader from one WAU stated that no monitoring issues had been identified during Watershed Analysis. In several others, interest was expressed by either the team leader or team members but action was not initiated to follow through with development of a monitoring plan.
Since development of a monitoring plan was voluntary, the decision of a team to develop a monitoring plan can be traced to the fact that somebody believed it was important. In some cases, monitoring was a priority for one of the organizations involved, so someone in authority provided direction to their representative on the team to develop a monitoring plan. In other cases, a team member that felt strongly about monitoring became an advocate for a monitoring plan and persuaded other participants to proceed.
A variety of reasons were cited for initiating monitoring. Participants wanted to a) evaluate the effectiveness of prescriptions, b) determine how resources (fish populations, fish habitat and water quality) would respond over time, c) collect baseline data that was missing in the initial analysis, d) produce information for use in the five-year review, and e) validate the Watershed Analysis process itself. In some cases, a monitoring component was needed because the WA was going to be used as part of a landowner Habitat Conservation Plan. Some landowners also viewed monitoring as a tool for cooperative resolution of resource concerns that could prevent or replace adversarial relationships.
Reasons why some WA teams did not initiate monitoring
In some cases development of a monitoring plan was not initiated because the team or team leader did not identify any compelling reason or benefit. Some teams appeared confident that the analysis and prescriptions were on target and had identified no monitoring issues or concerns. Many teams were unaware of a five-year review and had not considered how monitoring data would contribute to a meaningful and successful review process.
In other cases, a team or team leader interested in monitoring was simply too busy to initiate additional projects. The timing of monitoring plan development was a problem, because teams were typically overloaded tying up loose ends, and many people have to move on and begin another analysis or catch up on other work that had been neglected during the analysis. Occasionally, it appeared that conflict and suspicion had developed among participants during an analysis. This atmosphere discouraged participants from working together to develop a monitoring plan because they did not recognize that a monitoring plan could help overcome mistrust and resolve conflict.
What we learned
To motivate WA teams to initiate monitoring, it is necessary to:
- Demonstrate that WA monitoring produces something of value, that it can be done at a reasonable cost and that it won't go on forever.
- Increase awareness of the benefits of cooperative monitoring. It is particularly important to sell the concept to people at the policy level.
- Inform and motivate team members to advocate for monitoring in the absence of a policy from higher up.
- Increase awareness of the five year review and the importance of monitoring in making the review and evaluation process more constructive.
- Demonstrate how monitoring can help resolve conflicts and discomfort over uncertainty about the analysis.
- Overcome the fears about monitoring that cause people to reject the idea of monitoring without giving it a chance.
Step 2: Identification of monitoring goals and objectives
Five Watershed Analysis teams initiated the process of identifying monitoring goals and developing monitoring objectives. The teams used several approaches to accomplish this step.
Three of the five teams had completed prescriptions before attempting to identify monitoring objectives. Identification of monitoring goals and objectives was done through a group process by interested members of the prescription team, along with some assessment team analysts. All three teams identified a mixture of monitoring objectives, but some placed more emphasis on monitoring to evaluate the effectiveness of prescriptions while the other placed more emphasis on monitoring to document trends in resource conditions. All three teams were successful in identifying a set of monitoring objectives in group meetings. The time required for these groups to identify objectives ranged from 0.5 to 2 days, depending on how much supporting information was assembled during the group process.
Two other teams began working on monitoring plan development prior to completing prescriptions. One team attempted to identify monitoring goals and objectives while the prescription process was underway with a group that included a mixture of prescription team members and resource team analysts. Because the prescriptions were not available, the team had difficulty preparing monitoring objectives to evaluate the effectiveness of prescriptions. This problem was resolved by postponing further work on monitoring objectives until the prescriptions were completed.
The other team took a different approach. They made a commitment to develop a monitoring plan during start-up and designated a monitoring coordinator who began working during the resource assessment process. The monitoring coordinator identified a member of each resource assessment team to act as the monitoring contact person for their module and worked one-on-one with these people to identify monitoring objectives during resource assessment and synthesis. Potential monitoring objectives were identified during interviews with the module contact people and recorded by the monitoring coordinator.
Unfortunately, in both cases where teams initiated work on monitoring prior to prescriptions, finalization of prescriptions has taken months. Neither group has been able to finish identifying monitoring objectives. Hopefully, both teams will be able to pick up the lost momentum when prescriptions are finalized.
Several teams encountered situations that made identification of monitoring objectives and monitoring hypotheses more challenging. In WAUS with rapid urbanization and mixed patterns of urban, agriculture and forest land use, it was confusing to develop monitoring hypotheses about the effectiveness of WA prescriptions because of the difficulty in separating the effects of various land use impacts.
One Watershed Analysis was a joint state and federal effort that involved use of both state and federal guidelines. The two processes differ in purpose, procedure, content and spatial scale. Several differences are of particular importance in monitoring plan development. Federal WA has more modules, including cultural and wildlife modules that are not directly related to stream channels. The federal WA does not generate causal mechanism reports or prescriptions, making it more difficult to develop cause and effect monitoring hypotheses and to determine how to monitor effectiveness. The procedures for identifying monitoring objectives in the monitoring module seemed to work effectively for identifying monitoring objectives related to the federal cultural and wildlife modules.
Factors that contributed to successful development of monitoring objectives
- An organized process with clearly identified goals and tasks.
- A committed leader that kept the team on track and followed through to keep the process moving.
- A positive group atmosphere and the ability of the team to work well together. This usually carries over from successful teamwork during resource assessment and prescriptions process.
- Representation from both prescription and resource assessment team members to identify a broader range of monitoring objectives.
- Completed prescriptions were essential for developing monitoring objectives related to prescription effectiveness.
- A monitoring contact person that worked directly with assessment team leaders during resource assessment was effective in capturing monitoring objectives related to filling baseline information gaps and validation of causal relationships that could be lost later in the process.
Factors that hindered development of monitoring objectives
- Lack of committed leadership to sustain progress on the work.
- Lack of familiarity with monitoring concepts and terminology.
- A negative group atmosphere where conflict and mistrust between team members interferes with discussion of ideas and consensus decision-making.
- Fears and negative attitudes about monitoring that cause team members to shoot down potential monitoring objectives without examining whether their concerns are justified.
- Confusion caused by the lack of clearly focused monitoring goals, or the need to organize and prioritize large numbers of potential monitoring objectives.
- Inadequate participation of resource assessment team members and failure of assessment teams to document monitoring suggestions prior to disbanding.
- Lack of causal mechanism reports and prescriptions for the federal portion of joint state/federal Watershed Analyses.
- The long time-span required to finalize prescriptions in some WAUS breaks the continuity and momentum of the team before monitoring plan development can begin.
What we learned
- The TFW Ambient Monitoring program can assist teams by facilitating the group process and providing technical assistance, but leadership from the team is essential to make it a success.
- There is more than one way to successfully identify monitoring objectives. Teams need a combination of guidance and flexibility to succeed.
- Input from both resource assessment and prescription team members is needed because both groups tend to identify different types of monitoring objectives.
- Group ownership of the monitoring objectives is essential to ultimate success. This is created by an inclusive group process that fosters participation rather than a process driven by one organization or individual.
- Teams need to clearly understand the purpose of the monitoring objective identification process.
- The monitoring objective identification process should be well organized to be effective.
- The concept of identifying monitoring objectives is unfamiliar to most people, so more familiarity with the concepts prior to doing it would be helpful.
Step 3: Preparation of the monitoring plan report
The monitoring plan report includes the monitoring objectives and supporting information such as a monitoring hypothesis, potential monitoring parameters, prognosis, time line, and cost estimate. The main obstacle to completion of this step is the substantial amount of time required to compile this information. Several teams compiled a significant amount of supporting information during their initial group meetings, however it was not feasible to complete the monitoring plan report during the group meetings. The Kennedy team did not attempt to develop supporting information during the group meeting. Instead, the assignment of pulling together supporting information and a draft monitoring plan report for the Kennedy WAU was assigned to a team member who volunteered for the task. In the Stillman and Willapa the major landowner volunteered the services of a staff member to complete this task.
The Kennedy team completed a draft monitoring plan report which is being reviewed by the rest of the team members. The Stillman team did not present the monitoring objectives and supporting information in a separate monitoring plan report as suggested in the module. Instead, this information was combined with the detailed sampling plan in one document. There appears to be no disadvantage to combining both steps if the commitment and staff time is available to undertake both steps simultaneously.
Factors that helped preparation of the monitoring plan report
- A dedicated person with a significant amount of time to devote to preparation of the monitoring plan report.
- Access to resource assessment team leaders who can provide needed information.
Factors that hindered preparation of the monitoring plan report
- Inability of the person developing the plan to devote adequate time to this task due to other conflicting job duties and priorities.
- Lack of training and experience in developing monitoring plans.
What we learned
- Participants should not underestimate the amount of work involved in developing the monitoring plan report and should dedicate adequate staff time and resources to the job.
- The person writing the monitoring plan report should utilize knowledge and expertise available from other team members.
- Get as much supporting information as possible out of the meetings where monitoring objectives are identified.
- It is important to get feedback from the entire team prior to finalizing a monitoring team report to ensure that the monitoring objectives have been properly represented.
Step 4: Development of detailed sampling plans
Only one team (Stillman) has completed a draft version of a detailed sampling plan for their monitoring objectives. Completion of this task involves a substantial amount of time and the compilation of detailed technical information on sampling design, methods, quality assurance data analysis and interpretation.
What we learned
- Information on topics such as sampling designs, standard methods, quality assurance, data analysis and interpretation can be difficult to find. It would be helpful for the TFW Ambient Monitoring Program to compile relevant information into a reference library or document. This would reduce the amount of time expended searching around for this information.
- People with varying levels of experience can develop detailed sampling plans if they have access to specialists who can answer questions and provide detailed information.
- Gauging the time required to complete monitoring projects is the greatest challenge in developing cost estimates.
- Most organizations lack experience in developing sound sampling plans.
Step 5: Obtaining commitments to implement monitoring plans
None of the teams have completed securing commitments to implement their monitoring plans, although the major landowner in the Stillman WAU has committed to implementing a suite of monitoring objectives. Initial discussions have been undertaken in Kennedy and Sol Duc. In the Kennedy WAU, there has been preliminary discussions about the idea of developing a grant proposal fund implementation of some of the monitoring objectives. In the Sol Duc, the monitoring coordinator began communicating with a wide variety of organizations in the area to inform them about the development of the monitoring plan and to identify organizations that would potentially be interested in committing resources, volunteer time or in-kind services to the monitoring effort.
Factors that helped teams secure commitments to implement monitoring objectives
- Communication with a wide range of potentially interested groups in the WAU.
- Ownership among WA participants of the monitoring objectives, and understanding of the importance of monitoring information to the success of the WA.
- Ability to provide specific information on costs, time-line and expertise required.
- Ability to demonstrate that monitoring objectives are well defined, and that useful products will be produced.
Factors that hindered securing commitments to implement monitoring objectives
- Concern about costs.
- There is a perception that expending resources to monitor completed Watershed Analyses will reduce, resources available to conduct additional Analyses.
- Failure of WA participants to recognize and take advantage of resources within the broader community.
- State and federal budget cuts for resource management agencies that reduce staff and money available for monitoring.
What we learned
- It is important to begin communicating with community groups to build interest in WA monitoring early in the process.
- Reach out to community for support.
- Focus monitoring objectives and identify specific products and deliverables.
- Be creative in seeking sources of funding and support.
Conclusions and Recommendations
Through the pilot project we have identified four key elements to success. For monitoring teams, these elements often represent challenges that must be overcome. The four elements are:
- motivation to develop and implement a plan,
- an organized procedure to successfully develop and implement the plan,
- scientific and technical know-how, and
- adequate resources (personnel and money).
A more detailed discussion of these elements follows...
Motivation is the most important element. Since monitoring is voluntary, a monitoring plan will not be initiated, developed or implemented if participants are not motivated. To have the motivation necessary to follow through during the course of a monitoring project, participants must be able to identify benefits from monitoring that are worth significant expenditures of time, money and effort. Some people believe that monitoring will not produce useful information that can be used to improve resource management. This impression is often based on experiences with unsuccessful monitoring programs in the past. Other people may feel threatened by the unknown, or fear that monitoring results may require changes. Actual examples of successful Watershed Analysis plans are needed to demonstrate how monitoring has produced useful results and to inspire interest and support for monitoring.
- There is a need to develop information documenting why the development and implementation of a monitoring program is an important component of Watershed Analysis. This information needs to be provided to policy level people because policy support for monitoring is essential. Information should also be provided to the field managers and participants in Watershed Analysis.
- There is a need to increase awareness of the five-year review and to document the importance of monitoring information in a successful review process.
- There is a need to compile examples of successful monitoring and the benefits it has provided to participants.
Developing and implementing a monitoring plan is a complex task involving many steps that need to be done correctly and in the proper sequence. The ability to identify the multitude of tasks, and organize and deploy people and resources in an effective manner is a critical element to successful planning and implementation of monitoring efforts. Organizing and managing a project of this nature is a significant challenge, particularly for participants that do not have monitoring experience and may not be familiar with monitoring concepts or terminology, or have experience in monitoring, project management, or cooperative inter-organization endeavors.
The cooperative monitoring module of the Watershed Analysis Manual is primarily an organizational tool designed to help overcome this challenge. The monitoring module B attempts to identify various tasks that need to be accomplished, the types of information that is needed, and the personnel required. No fatal flaws in the monitoring module were identified during the pilot test. The flexibility provided to teams to adapt the procedure to their needs and style appears to be a critical element to the success of the module.
The support services provided by the TFW Ambient Monitoring Program included training of participants, facilitation of meetings, and technical assistance. We received favorable comments on these services from the teams we worked with. Support services appear to be a valuable method of helping teams overcome organization challenges, especially for participants that have not been involved in development of a monitoring plan.
- Refine the monitoring plan report to include an initial cost estimate and time-line for each monitoring objective.
- Conduct an annual survey of teams that have used the monitoring module. The summary should identify: a) changes that can be made to improve the process; b) evaluate the utility of the information assembled; and c) identify helpful tips and approaches to planning and implementation.
- Incorporate monitoring training in the WA training sessions, and develop training opportunities for analysts who went through the training sessions before the monitoring module was introduced.
- Provide on-site training and assistance for teams preparing begin work on monitoring plans.
- Continue to help facilitate the group process of identifying monitoring objectives.
Scientific and technical know-how
A successful monitoring program depends on the application of scientific knowledge and techniques to answer monitoring questions. It must provide clear monitoring hypotheses and methods of collecting and analyzing the data needed to prove or reject the hypotheses. Scientific methods that are repeatable and sampling designs with the power to detect change and distinguish trends from natural variation are useful to help overcome this challenge.
- There is a need to develop additional standard methods, especially methods for monitoring of input processes and triggering mechanisms.
- There is a need to develop information and training workshops on sampling design and data interpretation to help transfer this technical know-how into the realm of the practitioners that will be developing sampling plans.
Lack of adequate resources to develop monitoring plans or conduct monitoring activities is a major obstacle to success. The pilot project has demonstrated that development of a monitoring program alone is a significant undertaking requiring extensive time and effort. Monitoring occurs at a time when participants in the WA are often tired, burnt out, and need to give attention to other tasks that have been ignored during the analysis. It is important to recognize the time and effort required to develop a monitoring plan, and budget for it when planning for the Watershed Analysis.
Funding for conducting monitoring is a critical issue. Conducting Watershed Analysis is putting significant stress on the resources of many organizations, particularly governmental organizations faced with budget cuts. Some organizations perceive that they are faced with a choice of doing more analyses or conducting monitoring on the ones already completed. Motivation is a key factor in determining if resources to conduct monitoring will be made available. It is critical that monitoring is perceived as a integral and critical component of WA to receive the resources needed. If monitoring is viewed as an unnecessary add-on activity, it is unlikely to be done.
- There is a need to help identify and develop funding sources, such as grant programs, for monitoring plan implementation.
- There is a need to help establish linkages between monitoring teams with work to be done and college training programs, student interns and community groups that can provide a source of volunteer labor.
Thanks to the Washington Forest Protection Association for funding to conduct this project. Thanks to members of the Kennedy Creek, Sol Duc, Stillman, West Kitsap and Willapa Watershed Analysis teams for using the monitoring module and sharing their experiences with monitoring plan development.