Systems Engineering Applications for Small Business Innovative Research (SBIR) Projects

Transcription

1 Air Frce Institute f Technlgy AFIT Schlar Theses and Dissertatins Systems Engineering Applicatins fr Small Business Innvative Research (SBIR) Prjects Phillip J. O'Cnnell Fllw this and additinal wrks at: Part f the Systems Engineering Cmmns Recmmended Citatin O'Cnnell, Phillip J., "Systems Engineering Applicatins fr Small Business Innvative Research (SBIR) Prjects" (2012). Theses and Dissertatins This Thesis is brught t yu fr free and pen access by AFIT Schlar. It has been accepted fr inclusin in Theses and Dissertatins by an authrized administratr f AFIT Schlar. Fr mre infrmatin, please cntact richard.mansfield@afit.edu.

2 SYSTEMS ENGINEERING APPLICATIONS FOR SMALL BUSINESS INNOVATIVE RESEARCH (SBIR) PROJECTS THESIS Phillip J. O Cnnell Captain, USAF AFIT/GSE/ENV/12-S01 DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY AIR FORCE INSTITUTE OF TECHNOLOGY Wright-Pattersn Air Frce Base, Ohi DISTRIBUTION STATEMENT A APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED

3 The views expressed in this thesis are thse f the authr and d nt reflect the fficial plicy r psitin f the United States Air Frce, Department f Defense, r the U.S. Gvernment. This material is declared a wrk f the U.S. Gvernment and is nt subject t cpyright prtectin in the United States.

4 AFIT/GSE/ENV/12-S01 SYSTEMS ENGINEERING APPLICATIONS FOR SMALL BUSINESS INNOVATIVE RESEARCH (SBIR) PROJECTS THESIS Presented t the Faculty Department f Engineering and Engineering Management Graduate Schl f Engineering and Management Air Frce Institute f Technlgy Air University Air Educatin and Training Cmmand In Partial Fulfillment f the Requirements fr the Degree f Master f Science in Systems Engineering Phillip J. O Cnnell, MAS Captain, USAF September 2012 DISTRIBUTION STATEMENT A APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED

5 AFIT/GSE/ENV/12-S01 SYSTEMS ENGINEERING APPLICATION FOR SMALL BUSINESS INNOVATIVE RESEARCH (SBIR) PROJECTS Phillip J. O Cnnell, MAS Captain, USAF Apprved: Jseph R. Wirthlin, Lt Cl, USAF (Chairman) James C. Malas, PhD (Member) Sm R. Sni, PhD (Member) Date Date Date

6 Abstract The purpse f this research was t define Systems Engineering applicatins fr Small Business Innvative Research (SBIR) prjects. Specifically, this thesis sught t answer five research questins addressing the essential elements and applicatin f Systems Engineering prcesses within the SBIR cmmunity. Infrmatin was cllected frm multiple rganizatins thrughut the SBIR cmmunity t supprt this research. The research identified that current DD and Air Frce Systems Engineering Plicy d nt adequately address SBIR prjects and SE prcesses are nt well dcumented within the cmmunity. This research identified the need t tailr a Systems Engineering apprach fr SBIR prjects as verarching plicy is nt tailred fr SBIR. Results frm this wrk identified the SE tasks identified in Air Frce plicy. The culminatin f this effrt defined the current SE tasks in the SBIR cmmunity as well the verall SE rigr being applied in the different Systems Engineering Prcess areas identified in DD and Air Frce Systems Engineering Plicy. iv

7 Acknwledgments I wuld like t express my sincere appreciatin t my faculty advisrs, Dr. Sm Sni and Lt Cl Jseph Wirthlin, fr their guidance and supprt thrughut the curse f this thesis effrt. Their insight and experience was certainly appreciated. I wuld, als, like t thank my spnsr, Dr. James Malas, frm the Air Frce Research Lab fr bth the supprt and latitude prvided t me in this endeavr. Phillip J. O Cnnell v

8 Table f Cntents Page Abstract... iv Acknwledgments... iv Table f Cntents... vi List f Figures... ix List f Tables...xx I. Intrductin...1 Backgrund...1 Challenges...1 Past Research...2 Prblem Statement...3 Research Fcus...4 Methdlgy...4 Assumptins/Limitatins...5 Implicatins...5 Summary...5 II. Literature Review...6 Chapter Overview...6 Defense Acquisitin Guidebk...7 AFI AF SEAM...12 vi

9 AFRLI Plicy...14 Past Research...19 HELLTP...22 Deplyed Base Energy Alternatives Reprt...23 CGOIP...24 TASE...26 Case Study Summary...28 III. Methdlgy...29 Chapter Overview...29 Research Objectives...31 Research Questins...31 Hypthesis...31 Interview Instrument Develpment...32 Test Subjects...33 Summary...34 IV. Analysis and Results...35 Chapter Overview...35 Interview Results...35 Summary f Results...59 AFRL 8 SE Key Questins...63 AF SEAM Cmparisin...65 Research Questins Answered...66 vii

10 Summary...67 V. Cnclusins and Recmmendatins...68 SBIR SE Checklist...69 Significance f Research...71 Recmmendatins fr Actin...71 Recmmendatins fr Future Research...72 Appendix 1: SBIR SE Interview...73 Appendix 2: SBIR AF SEAM and AFRL Plicy SE Tasks...80 Appendix 3: SBIR SETT Tl Output...90 Bibligraphy...95 Vita...96 viii

11 List f Figures Figure Page 1. AFRL Prcesses Mapped t the SE Vee AFRL S&T SE HELLTP Tp Level IPPD Mdel CGOIP Streamlined SE Apprach CGOIP Prpsal Checklist Grunded Thery All Participants Results Technlgy Directrates Results Test Centers Results Air Lgistics Centers Results ix

12 List f Tables Table Page 1. SE Prcesses DAG Prcesses and Rles f the PM and SE DAG SE Prcesses AF SEAM SE Ttal Practices AFRL 8 SE Key Questins Mapped t the DAG Ttal Participants Interview Results Clr Scale Stakehlders Requirements Definitin Requirements Analysis Requirements Management Decisin Analysis Technical Planning Technical Assessment Techncial Data Management Risk Management Cnfiguratin Management Interface Management Architectural Design Implementatin Integratin x

13 21.Verificatin Validatin Transitin AFRL SE Key Questins Mapped t Interview Results SBIR SE Checklist...69 xi

14 I. Intrductin Backgrund The Air Frce Small Business Innvatin Research (SBIR) prgram is vital element f the Air Frce Research Labratry (AFRL) cntracts prtfli perated under the guidance f the Air Frce SBIR/Small Business Technlgy Transfer (STTR) Prgram Manager within AFRL at Wright Pattersn Air Frce Base. The SBIR prgram funds early-stage R&D prjects at small technlgy cmpanies that supprt a Department f Defense (DD) need and have the ptential fr cmmercializatin in private sectr and/r military markets. The DD s SBIR prgram is a large part f the multibillin dllar federal SBIR prgram administered by twelve federal agencies acrss the cuntry [ The DD and Air Frce prvide tp level Systems Engineering guidance and plicy fr the acquisitin cmmunity. Hwever the current guidance and plicy has nt yet been tailred specifically fr the SBIR cmmunity. Challenges The DD has well defined system engineering prcesses dcumented in the acquisitin 5000 series fr typical acquisitin prgrams. Hwever a number f challenges exist with applying Systems Engineering t SBIR prjects since they are unique cmpared t typical acquisitin prgrams. They are managed by many different small businesses that may r may nt have an rganic SE capability. Additinally they vary significantly in scpe, are small in size, shrt in prject length, and are early 1

15 research prjects. They are categrized as 6.1 Basic Research r 6.2 Applied Research prjects which are further defined in AFRLI Scientist and Engineer Manuel. Tpics are generated acrss the Air Frce by Prgram Executive Officers, Technlgy Directrates, Air Lgistics Centers and Test Centers. SBIR prjects are develped in three phases. Phase 1 is a technical feasibility study that allcates up t $150k and 9-12 mnths. Phase II is cncept develpment and allcates up t $1M and 24 mnths. There are als Critical Manufacturing SBIR prjects that are allcated up t $5M fr Phase II. Phase III is the cmmercializatin stage [ SBIR prjects managed by many different rganizatins thrughut the DD. Within the Air Frce, SBIR prjects are managed by AFRL Technlgy Directrates, Test Centers and Air Lgistics Centers. Systems engineering is a technical management prcess that can be used t help ensure that prjects are successfully implemented t the next phase f develpment if selected. As SBIR prjects vary cnsiderably in scpe, are managed by many different rganizatins within the gvernment, and wrk is accmplished by varying small businesses it presents a prblem f ensuring that cnsistent systems engineering prcesses are being applied acrss all prjects. Past Research AFIT past research effrts have helped t identify areas fr imprvement fr applying SE t the S & T cmmunity. Mst ntably a thesis cmpleted by AFIT called A Tailred SE Framewrk fr S & T Prjects develped a tailred systems engineering apprach fr typical S & T prjects. This wrk is discussed further in Chapter II as part 2

16 f the literature review. SBIR hwever is nt a typical S & T prject and AFRL has expressed interest in a similar prject that wuld define the SE rigr needed fr a SBIR prject and prvide a tailred apprach fr implementing SE prcesses int their SBIR prjects. Past research fr gverning this material is cvered in Chapter II. Prblem Statement Systems engineering prcesses are nt fully defined in plicy and are nt being implemented cnsistently and t adequate levels fr all SBIR prjects. SBIR is a unique prgram that challenges small and large business participants t wrk tgether. Small business participants may r may nt have SE principals as defined by the DD incrprated int their culture. Identifying the adequate level f SE and ensuring it is incrprated in a cnsistent manner acrss all rganizatins fr SBIR prjects will help t ensure prjects are ready t prceed t the next phase f develpment. This will aid in the future transitin f their prjects. Current DD r Air Frce plicy des nt specifically define SE prcesses fr SBIR prjects. AFRL has mapped their SE plicy t the Defense Acquisitin Guidebk (DAG) in AFRL Instructin Science and Technlgy Systems Engineering. Hwever applicatin f Air Frce Systems Engineering plicy has yet t be identified fr applicatin within the SBIR prgram. Current Air Frce plicy fr AFRL prgrams are utlined in AFRL Instructin , AFRL Manuel AFRL Scientist and Engineer Manuel and AF Instructin Life Cycle Systems Engineering. These 3

17 plicies are in alignment with the DD s Systems Engineering guidance captured in the DD 5000 series and the DAG. Research Fcus The fcus f this thesis is t identify hw current systems engineering practices apply fr SBIR prjects. This included identifying hw and what current DD and Air Frce SE plicy apply t SBIR prject during Phase I and II and hw t best tailr the guidance t develp a slid SE apprach fr the technical management f the prject. Thus this thesis fcuses n implementatin f early systems engineering prcesses fr Phase I and II SBIR prjects. Withut a slid SE apprach SBIR prjects are at risk t fail. Gd SE prcesses will help t ensure prjects better prepared fr prceeding t their next phase f develpment while adequately managing technical risk. Methdlgy Preliminary research included identifying relevant SBIR dcumentatin, past case studies and previus wrk. It was quickly identified that there was insufficient SBIR dcumentatin t supprt this apprach. Very little Systems Engineering dcumentatin was fund t be assciated with SBIR prjects and varied amng rganizatins. Thus it became essential t cnduct interviews t gather the infrmatin needed. Then interview and literature review data culd be analyzed using a triangulatin apprach t identify the relevant Systems Engineering applicatin f principles. 4

18 Assumptins/Limitatins N case studies that fcused n direct applicatin f systems engineering n SBIR prjects were fund. Since rganizatins managing SBIR prjects are gegraphically separated it is nt feasible t gather data frm enugh rganizatins t have valid data that represents all SBIR prjects. Therefre, this study is based n representative sampling. Implicatins Thugh this prject fcuses specifically n SBIR prjects, findings will likely be t similar S & T prjects. Prjects in early develpments will have many similar attributes t the SBIR prjects analyzed in this research. This wrk culd be used t guide a tailred SE apprach fr similar prjects/prgrams. Summary This chapter prvided an verview f research. Chapter II will review relevant literature. Chapter III will prvide an in depth lk at the methdlgy. Chapter IV will analyze data fr this research. Chapter V will prvide results and cnclusins fr this research. 5

19 II. Literature Review Chapter Overview The purpse f this chapter is t review past wrk accmplished n Systems Engineering in AFRL and identify current SE plicy as it pertains t the S &T Cmmunity. The Department f Defense has published the Defense Acquisitin Guide and the DD 5000 series t identify SE prcesses. The Air Frce has published Air Frce Instructin (AFI) Life Cycle Systems Engineering fr the acquisitin cmmunity. Additinally, Air Frce Material Cmmand develped the Air Frce Systems Engineering Assessment Mdel (AF SEAM) fr assessment f Air Frce prgrams. AF SEAM is a very useful SE assessment tl fr typical Air Frce acquisitin prgrams. Differences between the DAG, AFI and AF SEAM can be cnfusing since they vary in terminlgy. The belw graphic identifies the SE prcesses identified in each dcument. 6

20 Table 1: SE Prcesses (AF SEAM, Sept 2010) Defense Acquisitin Guidebk The Defense Acquisitin Guidebk (DAG) is designed t cmplement DD plicies identified in DD Directive The Defense Acquisitin System and DD Instructin Operatin f Defense Acquisitin System. Chapter 4 f the DAG cvers Systems Engineering. It cvers the system design issues facing a prgram manager, and details the Systems Engineering prcesses that aid the prgram manager in designing an integrated system that results in a balanced capability slutin [DAG, 2012]. 7

21 It defines Systems Engineering as: an interdisciplinary apprach and prcess encmpassing the entire technical effrt t evlve, verify and sustain an integrated and ttal life cycle balanced set f system, peple, and prcess slutins that satisfy custmer needs. Systems Engineering is the integrating mechanism fr the technical and technical management effrts related t the cncept analysis, materiel slutin analysis, engineering and manufacturing develpment, prductin and deplyment, peratins and supprt, dispsal f, and user training fr systems and their life cycle prcesses [DAG, 2012]. The DAG sectin discusses early Systems Engineering and emphasizes the imprtance f early SE during technlgy develpment. The DAG als defines the rle f the Prgram Manager and Chief Engineer illustrated in figure 2 belw. The DAG als separates the abve 16 SE prcesses int tw areas shwn in figure 3. Table 2: DAG Prcesses and Rles f the PM and SE (DAG Table 4.1.1T1, 2012) 8

22 Table 3: DAG SE Prcesses (DAG Table T1, 2012) AFRLI attempts t translate these prcesses frm the DAG fr the science and technlgy cmmunity. Als sectin f the DAG identifies the fllwing SE tasks relevant t the S & T cmmunity: Key Systems Engineering Activities During Technlgy Develpment Interpret User Needs; Analyze Operatinal Capability and Envirnmental Cnstraints Develp System Perfrmance (and Cnstraints) Specificatins and Enabling/Critical Technlgies and Prttypes Verificatin Plan Develp Functinal Definitins fr Enabling/Critical Technlgies/Prttypes and Assciated Verificatin Plan Decmpse Functinal Definitins int Critical Cmpnent Definitin and Technlgy Verificatin Plan Design/Develp System Cncepts, i.e., Enabling/Critical Technlgies; Update Cnstraints and Cst/Risk Drivers Demnstrate Enabling/Critical Technlgy Cmpnents Versus Plan Demnstrate System and Prttype Functinality Versus Plan Demnstrate/Mdel the Integrated System Versus the Perfrmance Specificatin Demnstrate and Validate the System Cncepts and Technlgy Maturity Versus Defined User Needs Transitin t Integrated System Design Interpret User Needs, Refine System Perfrmance Specificatins and Envirnmental Cnstraints 9

23 Develp System Functinal Specificatins and Verificatin Plan t Evlve System Functinal Baseline Evlve Functinal Perfrmance Specificatins int System Allcated Baseline The DAG als identifies the fllwing fr SBIR: Small Business Innvatin Research (SBIR) Technlgies Cnsistent with the directin f DD Instructin , the prgram manager (PM) shuld prepare a Technlgy Develpment Strategy (TDS) that apprpriately uses the SBIR prgram t develp needed technlgies, includes the use f technlgies develped under the SBIR prgram, and gives fair cnsideratin t successful SBIR technlgies. During TDS preparatin, the PM shuld ensure that the strategy addresses transitin f relevant SBIR technlgies and includes budgeting f fllw-n funds fr test, evaluatin, and integratin, as needed, t achieve the desired technlgical maturity. In additin, the PM shuld cnsider SBIR technlgies as candidates fr incremental and blck system imprvement initiatives as well as t address cmpetitive prttyping requirements, particularly at the subsystems and cmpnent levels. T effectively leverage SBIR, the PM review and ensure cmpliance with DD SBIR Phase III plicy guidance and shuld engage their prgram ffice, Prgram Executive Office, systems cmmand, prduct center, r DD Cmpnent SBIR prgram crdinatr fr assistance. The PM shuld als cnsult the DD SBIR prgram Web site fr nline resurces and infrmatin including a prgram descriptin, database f past awards and key pints f cntracts Small Business Innvatin Research (SBIR) Cnsideratin Cnsistent with the directin f DD Instructin , the PM shuld include SBIR and give fair cnsideratin t successful SBIR-funded technlgies in Acquisitin Strategy planning. Nte that SBIR fllw-n develpment and acquisitin (Phase III, nt funded with the SBIR set-aside budget) may be able t be pursued n a sle-surce basis withut further cmpetitin. Cmpetitin fr Phase I and Phase II awards (cntracts funded by the SBIR set-aside budget) satisfies all statutry cmpetitin requirements. SBIR Phase III cntract awards have SBIR status and thus must be accrded SBIR data rights. SBIR Phase III wrk may be pursued directly thrugh Phase III cntracts r encuraged thrugh subcntracts via incentives. T effectively leverage SBIR, the PM review and ensure cmpliance with DD SBIR Phase III plicy guidance and shuld engage their prgram ffice, Prgram Executive Office (PEO), systems cmmand, prduct center, r Cmpnent SBIR prgram crdinatr fr assistance. 10

24 Air Frce Instructin Life Cycle Systems Engineering AFI identifies SE as encmpassing the entire set f scientific, technical, and managerial effrts needed t cnceive, evlve, verify, deply, supprt, and sustain a rbust prduct, platfrm, system, r integrated system-f-systems (SS) capability t meet user needs. It currently defines 12 SE prcesses that were shwn earlier in Table 1. It als defines SE respnsibilities fr the prgram manager and engineers. It hwever des nt specifically address SBIR prjects. AFI is currently under revisin and is prjected t better align with AF SEAM and the DAG. The revised draft is prjected t align with AF SEAM s 10 SE prcesses. The DAG defines 16 SE prcesses as als illustrated in Table 1. This discnnect in plicy can be cnfusing fr prject managers and engineers trying t decipher hw plicy applies t their prjects and hw best t develp a slid SE apprach. The revised versin f AFI shuld reduce this significantly by eliminating the current differences in the AFI. The authr nted these changes in his review f the 2011 draft versins f the updated AFI Discussins t make the new versin AFI have been held but n decisin has yet been made whether the updated versin will be AFI r With these changes cming t the AFI rganizatins shuld be cnsidering these changes fr future plicy updates. Als, they shuld be ready t ensure that these SE prcesses are being executed prperly nce the new plicy is published. 11

25 Air Frce Systems Engineering Assessment Mdel (SEAM) The primary purpse f AF SEAM is t prmte the applicatin and use f standard SE prcesses acrss the AF and t imprve the perfrmance f these prcesses thrugh Cntinuus Prcess Imprvement [AF SEAM, September 2010]. AF SEAM is nt yet mandated acrss the Air Frce hwever it is used as a reprting tl in sme AF cmmunities. It is als prjected that the revised AFI will align with AF SEAM. AF SEAM identifies ten AF standard SE prcess areas and lists assciated gals fr each. Specific practices and generic practices are identified fr each area. Thse areas are seen belw in Table 4 alng with the number f practices fr each area. Table 4: AF SEAM SE Ttal Practices (AF SEAM, Sept 2008) As seen abve AF SEAM identifies 190 ttal practices. This suggests a significant SE effrt fr any prgram t implement AF SEAM. There are three different training mdules develped t supprt AF SEAM describe in further detail in sectin 7 f AF SEAM. My experience as a SE instructr has taught me that this is a gd rigrus tl 12

26 fr a majr acquisitin prgrams. Hwever it must be tailred t a smaller scpe t be value added fr a smaller prject since it requires a large manpwer effrt. This is significant fr SBIR prjects since their limited scpe and resurces require an even mre tailred apprach t be value added. AF SEAM was designed t facilitate use tailring. Nt tasks can be cded N/A and nt be assessed. Generic practices can als be mitted. This ability t tailr it t a specific prject still requires a cnsiderable effrt since s many f the task may nt apply fr S & T prjects. This is even mre s fr SBIR prjects. Additinally, discussins with AFRL Plans and Prgrams quickly identified that AF SEAM is a very rigrus tl fr implementing SE that is nt tailred t an apprpriate level fr AFRL prjects. It is nt tailred fr the S&T cmmunity. In its current cnfiguratin, as many f the 190 SE tasks may r may nt apply given the attributes f the AFRL prject r prgram implementing this as is des nt make sense fr the SBIR cmmunity due t the uniqueness f their prjects, limited resurces and limited value added t the prject. A mre tailred apprach is required. Analysis f interview results in chapter IV fr each SE prcess will identify what SE tasks are being implemented and what are fr the SBIR cmmunity. 13

27 Air Frce Research Lab Plicy The Air Frce Research Labs have tw main dcuments that prvide guidance fr Systems Engineering. The first is AFRL Instructin which specifically addresses Systems Engineering in the S&T envirnment. The secnd is AFRL Manual fr Scientist and Engineers. AFRL Instructin S&T Systems Engineering AFRL Instructin Science and Technlgy Systems Engineering prvides SE guidance fr all f the AFRL cmmunity. It is in alignment with DD and Air Frce plicy but tailrs it t the S&T cmmunity. It identifies Eight Systems Engineering Key Questins t guide and assess the SE health f a prject. The questins are: 1. Wh is yur custmer? 2. What are the custmer s requirements? 3. Hw will yu demnstrate yu have met the requirements? 4. What are the technlgy ptins? 5. Which is the best apprach? 6. What are the risks t develping the selected technlgy? 7. Hw will yu structure yur prgram t meet requirements and mitigate risk? 8. What is yur business-based transitin plan that meets custmer apprval? AFRLI maps these questins back t the Systems Engineering Vee identified belw frm the Defense Acquisitin Guidebk. 14

28 Figure 1: AFRL Prcesses Mapped t the SE Vee (AFRLI , 2008) AFRLI Attachment 1 als maps the Eight SE Key Questins back t the DAG SE areas which is illustrated belw in Table 5. 15

29 Table 5: AFRL 8 SE Key Questins Mapped t the DAG AFRLI Attachment 1 prvides a questin and answer matrix fr the Eight SE Key Questins. It breaks the Eight SE Key Questins dwn t further detail, identifies What the Prgram Manager shuld knw abut his r her prgram and defines the clr assessment basis. It als identifies that Use f the key questins during reviews f basic research prgrams is ptinal. It des this separately fr 6.1 Basic Research, 6.2 Applied Research, 6.3 Advanced Technlgy Develpment, Advanced Technlgy Demnstratin and Manufacturing Technlgy. Als in Attachment 1, AFRL translates the 16 SE DAG prcesses fr the AFRL cmmunity. It defines the DAG prcess, defines the prcess fr AFRL and explains the imprtance. Review f this attachment identified that it is tailred fr the typical AFRL prgram and nt tailred fr SBIR prjects specifically. 16

30 Additinally AFRLI is currently being revised. The AFRL Systems Engineering Cuncil is currently reviewing the dcument. The new versin is much shrter and uses an AFRL Systems Engineering Guidebk t cmpanin the dcument. The guidebk details hw t implement SE prcesses int a prgram r prject. Bth the instructin and guidebk define the S&T SE Prcess in the belw illustratin: Figure 2: AFRL S&T SE Prcess (Draft AFRL SE Guidebk, 2012) This is similar t the SE streamlined prcess identified in the case study review. This prcess was successfully implemented in past case studies. Each step is explained in further detail t identify what SE tasks shuld be perfrmed. The guidebk als indentifies the Eight SE Key Questins and explains what shuld be dne fr each as indentified in the current instructin. 17

31 AFRL Manuel AFRL Scientist and Engineer Manuel The intent f this manual is t enhance DD Directives and Air Frce Instructins by placing actins int a chrnlgical prcess flw specifically develped fr the management f AFRL Wrk Units. Thus the manual prvides guidance n hw t technically manage wrk units within AFRL. The manual identifies SBIR as a three phase cngressinally mandated prgram established t stimulate technlgical innvatin, use small businesses t meet federal R&D needs, increase innvative, private sectr R&D cmmercializatin, and t encurage minrity and disadvantaged persns t participate in technlgical innvatin [AFRLM , 2003]. It identifies the different phases, funding and duratin. It als defines the SBIR schedule frm initial tpic call t phase cmpletins. The manual defines 6.1 Basic Research as a systematic study directed tward greater knwledge r understanding f the fundamental aspects f phenmena and f bservable facts withut specific applicatins twards prcesses r prducts in mind. It als defines 6.2 Applied Research as a systematic study t understand the means t meet a recgnized and specific natinal security requirement [AFRLM , 2003]. Mre infrmatin fr bth research categries can be fund in the manual. 18

32 Plicy Summary Review f Systems Engineering plicy within the DD, Air Frce and AFRL has identified very rigrus defined SE guidance and instructin. AFRL has dne a gd jb tailring their plicy t be in alignment with higher level guidance. Additinally AFRL s System Engineering Cuncil cntinues t be practive in the develpment f the new SE Guidebk fr better implementing gd SE prcesses within AFRL. Hwever current plicy at all the reviewed levels is nt specific enugh fr the SBIR cmmunity. Since SBIR is unique in many aspects as previusly discussed SE guidance needs t be better defined and tailred fr the SBIR cmmunity t ensure it is being implemented successfully. SBIR prjects are at risks t nt incrprate adequate SE prcesses withut better guidance. Additinally any future effrts t tailr SE prcesses fr the SBIR cmmunity shuld be in alignment with AF SEAM prcesses as the revised AFI is prjected t align with AF SEAM. Past Research Several effrts including AFIT graduate thesis wrk and past studies have been accmplished t analyze Systems Engineering effrts within AFRL. The authr identified mst ntably a past research prject A Tailred SE Framewrk fr S & T Prjects authred by Maj Pitzer, Maj Behm and Jane White that captured the Systems Engineering tasks and rigr fr typical AFRL prjects. They develped a tl called the Systems Engineering Tailring tl fr Science & Technlgy Prjects that defined prjects by 6 parameters: 19

33 1. RDT&E Categry (6.1, 6.2 r 6.3) 2. Prject Budget (less than $500k, $500k-$2M, greater than $2M) 3. Cre Prcess (CP-1,2 r 3) 4. Technlgy Readiness Level (1 thru 9) 5. Integratin Level (subsystem, system r missin) 6. Requirements Maturity (Technlgy Push r Requirements Pull) The tl then utputted what SE best practices (mapped frm the 16 DAG prcesses) that wuld apply t that prject/prgram. This tl is ntably similar t AF SEAM hwever it tailrs the tasks fr a prject based n the stated parameters. The SETT tl prvides a gd initial baseline hwever SBIR prjects are unique as previusly identified. Preliminary analysis f the SETT Tl identified that it als des nt tailr specifically t the SBIR cmmunity. Like AF SEAM, SETT identifies many tasks that may nt be t a specific SBIR prject due t its unique attributes.. Implementing a prcess r tl that is nt tailred t the apprpriate level risks creatin f nn value added wrk and can drain valuable resurces frm a prject. Bth the SETT Tl and AF SEAM are gd baselines t cnsider when identifying what SE tasks may apply t SBIR prjects. Typical parameters fr a SBIR prject inputted int the SETT Tl t establish a baseline are illustrated in Appendix 3. Additinally the authr identified fur ntable AFRL studies that were significant fr this research: 1. High Energy Laser On a Large Tactical Platfrm (HELLTP) 2. Deplyed Base Energy Alternatives Reprt 3. Cmpany Grade Officer Initiative Prgram 4. AFRL Transfrmatinal Activities in Systems Engineering (TASE) Assessment Phase Final Reprt Findings frm

34 The first tw studies fcused n the successful tailring and streamlining f SE effrts n tw larger AFRL prjects. The third study listed fcused n tailring and streamlining SE effrts t smaller prjects within AFRL as part f CGOIP. This study was very interesting since the prjects were being managed by CGOs with limited SE backgrunds. And like the first tw studies listed, CGOIP was als very successful in implementing gd SE prcesses int their prjects using a streamlined SE apprach. The last study fcused n making AFRL research prgrams mre effective and efficient, and imprving the transitin f technlgy t the warfighting cmmunity thrugh the use f gd SE prcesses. A number f very interesting findings were dcumented in this reprt. The studies selected fcused n S&T prjects that successful implemented SE prcesses. The gal was t establish a successful baseline frm histrical examples that define the SE rigr needed in the S&T cmmunity. The case studies fcused n prjects that frmed a multi-disciplinary team and implemented a tailred streamlined SE apprach fr their prjects. This apprach prved t be very successful in implementing gd SE prcesses fr the prjects. This apprach culd be very beneficial fr SBIR prjects if tailred t the apprpriate level. The fur studies identified were analyzed fr SE artifacts that cntributed t their success. 21

35 High Energy Laser On a Large Tactical Platfrm (HELLTP) The initial phase f this prject implemented the Air Frce s Integrated Prduct and Prcess Develpment (IPPD) prcess. The prject was cnsidered a Multidirectrate SE Initiative. It had three main bjectives: 1. Apply the IPPD prcess t the selected High Energy Laser n a Large Tactical Platfrm (HELLTP) prblem acrss multiple directrates, with custmer invlvement. 2. Assess the tls and prcess in the curse f executing the prgram. 3. Capture lessns learned with cmments and recmmendatins fr ging frward in the Phase II prgram. The prject was able t establish a successful team framewrk thrughut the IPPD prcess. As a result the team was able t tailr their SE apprach fr the prject. The belw figure illustrates their apprach. Figure 3: HELLTP Tp Level IPPD Mdel 22

36 The team identified the need t tailr their apprach as well as tailr the SE tls fr the prject. They classified their tls int fur classificatins: Preliminary SE, Requirements Management and Evaluatin, System Architecture Tls, and Mdeling and Simulatin. Frm there they were able t select the tailred tls needed fr their prject. Additinally, the team relied n a SE cntractr t prvide just in time training t the team. As part f the IPPD prcess the team cnducted reccurring meetings t access prject status and prgress. Deplyed Base Energy Alternatives Reprt The study fcused n applicatin f Systems Engineering principles t imprve technlgy investment utcmes. The prject elected t use a cntractr develped methd called Systems Engineering Tailred fr Science and Technlgy (SETFST). The fllwing steps were implemented: 1. Establish the study team (IPT) and define the verall prgram bjectives. 2. Define Desirements with team. 3. Generate alternatives. 4. Scre alternatives. 5. Exercise a value analysis mdel, and priritize the alternatives. Frmal team meetings were held with the IPT t review prject status. All key stakehlders were represented with membership n the IPT. The team develped a defined technical apprach fr the prject. As a result the team successfully implemented the SE prcesses identified in the SETFST methd selected. 23

37 Key findings: Review f the reprt clearly identifies that success f the prject was in direct relatinship with the success f the IPT. The team cnstruct ensured that they had the right mix f expertise fr the prject as well as team members with the expertise in Systems Engineering t successfully implement the prcess. This technical apprach t the prblem was very successful. The team implemented the SERFST methd which is cnsistent with DD and AF plicy. The SETST methd is similar t the streamline S&T prcess utlined in current AFRL plicy and is being incrprated int the AFRL SE Guide t cmpanin the revised AFRLI Cmpany Grade Officer Initiative Prgram (CGOIP) AFRL/RX pilted the Cmpany Grade Officer Initiative Prgram (CGOIP) in an effrt t streamline and tailr the SE effrt fr their prjects. The prjects were managed by CGO s that had minimal experience in SE. These prjects had an emphasis n transitin their prjects. The prjects were successful in implementing SE int the prjects by using the 5 step Streamlined SE Apprach and Prpsal Checklist illustrated belw. 24

38 Streamlined Systems Engineering Apprach Step 1 Step 2 Step 3 Step 4 Step 5 Frm Team Determine Requirements Generate Alternatives Evaluate Alternatives Dcument Results Update & check fr cmpleteness Exit Criteria fr all requirements - Perfrmance - Affrdability - Prducibility - Reliability - Supprtability Priritize & Validate with custmer Team brainstrm different slutin appraches Analysis f Alternatives - Cmpare Alt. Slutins with Reqts & Exit Crit Tech Readiness Assessment Manufacturing Readiness Assessment Risk Analysis Custmer Apprval f Sln USAF Prblem / Gal Custmer(s) SE Team Members List f Priritized Reqts & Exit Criteria Descriptin f Alternative Slutins TRA and MRA Risk Assessment Value Analysis 8 Figure 4: CGOIP Streamlined SE Apprach CGO IP Prpsal Checklist USAF Prblem / Gal Custmer(s) and User(s) IPT Members List f Requirements, KPPs, and S&T Exit Criteria - Objectives and Threshlds Alternative Slutin Appraches TRA and MRA Risk Assessment - Identificatin and Mitigatin Value Analysis fr Selecting Best Apprach In-huse Wrk Tasks Materials r Manufacturing Technlgy Related - Ratinale fr why is RX ding this Test Plan Prpsed Cst & Spend Plan Schedule with Majr Milestnes Technlgy Transitin Strategy Prgram Requirements Evaluatin f Alternatives Prgram Plan Figure 5: CGOIP Prpsal Checklist 25

39 Fllwing this streamlined apprach enabled the teams t successfully implement SE int their prcesses. As a result their prgrams were successful. This methd was similar t the streamlined SE prcess used fr case study 2. AFRL Transfrmatinal Activities in Systems Engineering (TASE) Assessment Phase Final Reprt Findings frm 2006 The gal f the prject was t make AFRL research prgrams mre effective and efficient (imprve S&T prgram perfrmance), and imprve the transitin f technlgy t the warfighting cmmunity (imprve technlgy transitin). The team fund a number f interesting findings as seen belw. Main trends discvered: AFRL prgram persnnel already have guidance n sund Systems Engineering practices (AFRLI , Science and Technlgy (S&T) Systems Engineering (SE) Initiative). Hwever, this guidance has sme shrtcmings. AFRL shuld use the Defense Acquisitin Guidebk (Chapter 4) as a framewrk fr imprving its Systems Engineering guidance because it is cmplete frm a prcess viewpint and is supprted by DD (the frmer USD/AT&L; nw the SecAF). Very few AFRL technlgy prgram leads fllw the AFRL Instructin r a cmplete set f Systems Engineering prcesses. This has led t prblems in requirements management, risk management, and ther areas that smetimes result in pr prgram perfrmance (including schedule/cst verruns) and transitin. Systems Engineering is nt freign t AFRL persnnel. Althugh it is nt widespread r cnsistently practiced, institutinalizing Systems Engineering prcesses shuld nt be as difficult as if they were cncepts new t AFRL. The team als discvered that: 26

40 Sme AFRL persnnel are cncerned that Systems Engineering prcesses are fcused n acquisitin (as ppsed t research) prgrams and might stifle the creative atmsphere essential t the discvery f new technlgies. AFRL has a requirement t implement rbust Systems Engineering prcesses in supprt f the DD and AF acquisitin prcess. DD has recmmended a series f best practices fr Systems Engineering (described in Chapter 4 f the Defense Acquisitin Guidebk). Current AFRL Systems Engineering guidance (AFRLI ) is nt adequate t ensure such a Systems Engineering prcess. In additin t nt being implemented by mst prgrams, it des nt address a sufficient number f Systems Engineering sub prcesses. Mst Systems Engineering practices are represented smewhere amngst the set f prgrams and ATDs assessed, s the cre understanding f gd Systems Engineering practices exist tday in pckets thrughut AFRL. ATDs and ther prgrams are mst successful when they have bth strng initial prcesses (requirements develpment and decisin analysis) and nging prcesses t address requirements changes and risk. Integrated Prduct Teams (IPTs) that include all stakehlders are essential. Prgrams have the mst difficulty with transitining technlgy t acquisitin custmers and warfighting users. This is due in part t changes in custmer pririties and funding Uniqueness in the way AFRL perfrms S&T prgrams lies nt in what they are develping r hw they develp technlgies; rather AFRL s unique nature lies in hw it fcuses its energies n the frnt and back end f the Systems Engineering prcess, making much f the intermediate functins the respnsibility f the cntractr. The Technlgy Directrates have many best practices that can be used by the rest f AFRL. This prject fcused n S&T prjects within AFRL hwever all f these findings d apply t the SBIR cmmunity within AFRL. They identify the risk f incnsistent applicatin f Systems Engineering and failure t fllw best practices. The findings 27

41 als identify that current SE plicy is nt adequate t ensure gd SE prcesses are being fllwed. This validates further the need t develp a tailred apprach fr the SBIR cmmunity since SBIR is even mre unique then typical S&T prjects. Case Study Summary The past research indentified successfully implemented SE prcesses fr their prjects. The studies had these key SE attributes: Frmed a multi-disciplinary team that invlved all relevant stakehlders Held team reviews t mnitr prject prgress Successfully tailred their SE apprach using a streamlined S&T prcess tailred t their prject that was cnsistent with Air Frce plicy In additin the TASE reprt validated the need t develp a tailred SE apprach fr the SBIR cmmunity as it identified many weaknesses in the S&T cmmunity and plicy fr gd implementatin f SE prcesses. It highlighted that current plicy is nt sufficient with AFRL and that varying levels f SE are being implemented. The reprt als nted that AFRL relies heavily n the cntractr t cmplete many SE tasks as is true with SBIR thus making it unique. Overall the literature review identified many pertinent SE prcesses fr the SBIR cmmunity as well sme f the struggles within the S & T cmmunity t fully integrate gd SE prcesses. 28

42 III. Methdlgy Chapter Overview The purpse f this chapter is t develp the methdlgy fr analysis. Fllwing the tpic selectin fr this research the authr began his literature review. The first step was t review all past research, SBIR dcumentatin and SE plicy. Several past research effrts n AFRL prjects were identified t be relevant. Figure 6 illustrates the apprach that was develped t gather and analyze data. Preliminary review f AFRL SBIR prjects quickly identified varying degrees f SE dcumentatin amng the different directrates. In mst cases there was little if any SE dcumentatin. Within AFRL prjects are required t submit a Frm 2913 Labratry Management Review at the beginning, annually and end f a phase. Sme directrates require the prject managers t answer the Eight SE Key Questins identified in AFRL plicy as an attachment. A clr scale was used t assess each questin. Out f apprximately tw dzen reviewed all f them shwed a green status fr all tasks. N additinal cmments were dcumented fr any f them. Prject Managers were nly required t prvide cmments fr yellw and red status questins. These became small vignettes hwever it was evident that additinal data wuld need t be cllected. Other dcumentatin fr SBIR includes prpsals, cntracts and final reprts. SE was fund t nt be well dcumented fr SBIR prjects within AFRL. The lack f SBIR SE dcumentatin identified that additinal data wuld be required. An interview instrument t cllect data frm SBIR prject managers was 29

43 develped and participants frm different rganizatins were identified. This apprach is illustrated belw: Tpic Literature Review SE Plicy & Guidance Review SBIR Dcumentatin Review Identify Participants Develp Interview Instrument Cnduct Interviews Analyze Results Output Findings Figure 6: Grunded Thery Once all data had been cllected results were analyzed t identify the SE tasks fr SBIR prjects. This research is an explratry wrk using a mixture f qualitative and quantitative methds in additin t using existing written materials as evidence. With the help f the SBIR prgram ffice ideal participants managing r verseeing SBIR prjects were identified fr the interviews. The interview is semistructured s that pen-ended respnses were encuraged and snwball sampling culd ccur. Using data gathered frm the literature review and interviews enabled a structured cntent analysis thrugh triangulatin t define data and the SE rigr that is assciated with SBIR prjects. Using a triangulatin analysis apprach enabled the authr t imprve the validity and reliability f this research [Glafshani, 2003]. 30

44 Research Objectives The bjective f this research prject is t define the SE rigr that shuld be best applied fr SBIR prjects. T define the rigr and design a tailred apprach will require identifying what degree f SE is in the SBIR envirnment and hw thse SE prcesses vary amngst prjects with respect t prject maturity, size and ther factrs. Research Questins T define the research bjectives the fllwing questin must be answered: 1. Hw well d SBIR prjects currently implement Systems Engineering plicy? 2. Hw are SE plicies implemented? 3. Hw d DD and Air Frce SE prcesses apply t SBIR prjects? 4. T what level f rigr des each SE prcess apply t SBIR prjects? 5. What is the best way t implement these prcesses? Hypthesis Different rganizatins implement varying levels f Systems Engineering int their SBIR prjects. Organizatinal SE plicies and SE knwledge base vary amngst SBIR prject managers. Therefre, prjects are at risk t fail meeting DD and AF standards with implementatin f SE and SE prcesses. This research is ging t test r measure the degree t which SE prcesses are applied t SBIR prjects. Then I will analyze the results t determine cmmnalities and differences between rganizatins. Frm this I hpe t generalize wrking SE principles fr the SBIR cmmunity. 31

45 Interview Instrument Develpment As identified in the literature review, AF SEAM fully defines the SE prcess invlved in a majr acquisitin prgram. Hwever these prcesses are defined differently in the DAG and AFI as illustrated earlier in Table 1. AFRL plicy maps the DAG prcess t their Eight SE Key Questins. The interview was created with this understanding and is designed t map directly back t AFRL plicy that maps t the DAG. One questin fr each SE prcess utlined in the DAG was created with the specific tasks called ut in AFRL plicy identified. The interview instrument is attached in Appendix 1. Additinal questins were asked abut the effectiveness f the Eight SE Key Questins utlined in AFRL plicy. AFRL identifies that the Eight SE Key Questins guide prject managers in implementing the SE prcess. The areas targeted during the interview were: 1. Jb, Organizatin, Experience, APDP Educatin, SBIR Experience fr demgraphic analysis 2. Stakehlders Requirements Definitin 3. Requirements Analysis 4. Requirements Management 5. Decisin Analysis 6. Technical Planning 7. Technical Assessment 8. Technical Data Management 9. Risk Management 10. Cnfiguratin Management 11. Interface Management 12. Architectural Design 13. Implementatin 14. Integratin 15. Verificatin 16. Validatin 17. Transitin 18. Usefulness f the AFRL Eight SE Key Questins 32

46 Test Subjects/Sample Size The day t day management f SBIR prjects fr the gvernment is typically perfrmed by a single SBIR prject manager. These prject managers are engineers r prgram managers with varying levels f experience. In sme cases the prject manager may have little experience wrking in the gvernment and even less experience with systems engineering. In ther cases they have been wrking in the gvernment fr 30+ years. Additinally Chief Engineers fr the directrates versee the management f these prjects. The ideal participants fr the interview were identified as SBIR Prject Managers, Engineers and Chief Engineers because they are the mst familiar with the daily technical management f a SBIR prject. The prject gal was t interview the Chief Engineer and several prject managers/engineers frm each rganizatin. Participants fr the interview were identified thrugh purpseful and snwball sampling. Interviews started with tw lcal directrates that were very supprtive f this SE research prject t establish a baseline. Additinal interview participants were selected frm multiple rganizatins t represent the brader SBIR cmmunity. The fllwing AFRL Technlgy Directrates participated: Materials Directrate, Prpulsin Directrate, 711 Human Perfrmance Wing, Space Vehicles Directrate and Munitins Directrate. Organizatins utside f AFRL that manage SBIR prjects that participated included: Hill AFB Rbins AFB Air Lgistics Centers, Arnld AFB Test Center and WPAFB Aernautical Systems Center. 33

47 Summary Analysis f the interview data and SE artifacts discvered in the literature review will allw identificatin f SE prcesses and best practices within the SBIR cmmunity using a triangulatin methd during analysis. Using this methd will help t validate results frm the different surces. The results cllected frm the interviews will als help t gauge the level f SE rigr being implemented within the different rganizatins managing SBIR prjects and help t identify the level f rigr needed fr SE tasks fr SBIR prjects. Cmparing thse results with current guidance and plicy discussed in Chapter II will identify the current Systems Engineering gap in plicy fr the SBIR Cmmunity and allw the authr t cmpare and cntrast the current plicy with the research findings t identify and develp a tailred SE apprach fr the SBIR cmmunity that aligns with DD and Air Frce guidance and plicy. 34

48 IV. Analysis and Results Chapter Overview This chapter captures results frm the interviews cnducted. It prvides a cnslidated representatin f results as well a detailed analysis fr each SE prcess area. Using results frm the interview prvides a SBIR cmmunity analysis f the Eight SE Key Questins and it answers the research questins utlined in Chapter III. Interview Results Interviews cnducted varied in size frm ne t several participants. The data was cllected by interviewing each rganizatin separately. AFRL XP helped t identify ptential survey participants fr each rganizatin. Participants selected were SBIR Prject Engineers/Prgram Managers and Chief Engineers. The fllwing number f participants represented each rganizatin and the interview results clr scale is seen belw: Table 6: Ttal Participants Materials & Manufacturing Technlgy Directrates Prpulsin Space Vehicles Test Centers Air Lgistics Centers Other Human Effectiveness Munitins Arnld Rbins ASC

49 Table 7: Interview Results Clr Scale Usually nt accmplished Smetimes accmplished Usually accmplished Almst always accmplished Usually nt accmplished: Less than 25% participants identified it as Smetimes accmplished: 25-50% f participants identified it as Usually accmplished: 50-75% f participants identified it as Almst always accmplished: 75% r greater f participants identified it as When interpreting the belw results make sure t cnsider that the specific SE tasks were derived frm AFRL plicy. Air Lgistics Centers and Test Centers have a different fcus. Thus sme f the tasks in the interview are in S & T terms which may nt apply in sme areas as they are written fr the ALC s and Test Centers. The additinal cmments dcumented fr each questin may better represent the SE tasks currently being accmplished in thse cases. 36

50 Stakehlders Requirement Definitin Overall results fr Stakehlders Requirements Definitin were very psitive. Mst f the participants identified with the tasks defined in AFRL Plicy. ALCs hwever did nt identify as well with these tasks. Stakehlders Requirement Definitin Table 8: Stakehlders Requirement Definitin All inputs frm relevant stakehlders translated int technical requirements. Requirements made quantifiable, have unique definitins, and specified threshlds and bjectives. Wrk with the user t establish and refine gals, attributes, perfrmance parameters, and financial and schedule cnstraints, and then ensure that all relevant requirements are addressed during the science and technlgy effrt. Translate the custmer needs int S&T prgram and system requirements. Ntewrthy r Significant Cmments Technlgy Directrates 37 Test Centers ALCs Ttal 71% 100% 33% 70% 71% 100% 67% 75% 71% 0% 33% 60% 79% 100% 33% 75% Technlgy Directrates stated: Review PEO/TEO Needs with WBS managers and Chief Engineers. Evaluate cst, feasibility (are the users requirements attainable and can the cmpany actually deliver the prduct with their rganic capability?) and the technlgy maturity level. Wrk t refine requirements by interacting with the spnsr. Identify likely transitin and what requirements are needed t give the tech a chance at transitin Test Centers stated: We attempt t include ther gvernment agency requirements if (ie Edwards AFB / NASA Aernautics/ etc. In general we d all f these, hwever we d nt include schedule cnstraints int ur planning. Schedule estimates fr S&T effrts are t uncertain t include

51 them in ur planning. Als, mst f these are really nly t the Phase II and beynd effrts. Phase I is t shw us yur capabilities, whereas Phase II is where we really drive technical requirements. Air Lgistics Centers: Develp/lead technlgy develpment/transitin teams t navigate R&D t implementatin. Our requirements are started in huse as we perfrm the maintenance n all their assets. Our prjects are based n reducing ttal wnership csts frm what we see in the field. Even thugh mst inputs and cmments identified with these tasks abut 30% f participants did nt identify with the SE tasks identified. Spnsr invlvement and requirements definitin t ensure valid requirements are being derived t meet the peratin need is an essential part f acquiring a successful system. Thus all participants shuld have identified with these tasks. Better educatin f SE principals wuld likely help cmmunity t better identify with imprtance f these tasks. Requirements Analysis Technlgy Directrates identified well with the identified tasks. ALCs and Test Centers did nt relate t the tasks as written in AFRL language. Als the scpe f SBIR prjects can vary cnsiderably frm lcatin s the task wrding my nt seem as phrased in sme cases. 38

52 Table 9: Requirements Analysis Requirements Analysis Obtain sets f lgical slutins t imprve understanding f the defined requirements and the relatinships amng the requirements (e.g. functinal, behaviral, tempral). Perfrmance parameters and cnstraints allcated and derived technical requirements defined. Partitin the technical prblem int selfcntained, chesive, lgical grupings f elements and, where apprpriate, defined the key interfaces. Technlgy Directrates Test Centers ALCs Ttal 71% 0% 33% 55% 86% 0% 100% 80% 50% 0% 33% 40% Ntewrthy r Significant Cmments Technlgy Directrates stated: Dn t identify a slutin, just help with prviding the expertise. Sunds a bit ivry twer (nt the way things really wrk). We lk at each prject n its wn relative merits. We have t make an inexact mental calculus n the impacts a successful prpsal culd have in addressing (1) primary spnsring custmers needs, (2) brader needs f USAF. We typically wrk with these same custmers thrughut the prgram and try t maximize relevance. And it s mre cmplicated than that. Smetimes we have t wrk with requirements in a mre diffuse manner. A new rad-hard memry chip fr example, may nt have a direct cnnectin t requirements at the custmer level, but trickle dwn thrugh specific develpments invlving memry chips that wuld benefit. We wuld never get that frm a direct custmer. This is the essence f being a technical expert in a labratry rganizatin and wrking acrss a lnger tempral perspective. Try t get the mst capability that can reasnably be expected. Air Lgistics Centers stated: Map the R&D int self-cntained, chesive, lgical grupings which can be incrementally funded seek funding. Our primary fcus is reducing the maintenance csts, ttal wnership csts. 39

53 Respnses fr these SE tasks were mixed. These tasks were prved t be mre t the Technlgy Directrates. Gd Requirements Analysis is an essential part f any system t ensure that requirements map back t valid perfrmance parameters and cnstraints. Results identify that the task Perfrmance parameters and cnstraints allcated and derived technical requirements defined is valid fr SBIR prjects. Requirements Management Overall the majrity f participants identified with the listed tasks fr Requirements Management. Hwever the ALC s and Test Centers did nt in all cases. SBIR phases are shrt in duratin and requirements typically t d nt frmally change during an early phase which likely accunts fr sme f the negative respnses. Table 10: Requirements Management Requirements Management Maintain the traceability f all requirements frm needs Dcument all changes t thse requirements Recrd the ratinale fr thse changes. Traceable t sme current r ptential future military capability need. Technlgy Directrates Test Centers ALCs Ttal 79% 100% 100% 80% 64% 0% 33% 50% 64% 0% 33% 50% 71% 100% 33% 70% 40

54 Ntewrthy r Significant Cmments Technlgy Directrates stated: I d all f these, but d nt dcument. Just keep in my mind and ntes f prgress f prject and hw the prject is leaning twards meeting an applicatin requirement, etc. Spnsrs are invlved with requirement changes. Increase in scpe dne thrugh interacting with spnsr and cntractr. Letters and are used fr dcumenting small changes. Cntract changes are required fr significant deviatin. It is still an inexact calculus. Custmer needs are nt necessary sufficiently precise fr the exercises yu believe happen in requirements management. In a Phase 1 fr example, the time frame is almst like an impulse functin (a single snapsht in time), like less than ne fiscal year. Only Phase 2 prjects have a gestatin interval lng enugh t matter in terms f evlving needs. It is a judgment call at that pint, and we find sme Phase 2 prjects are / are nt flexible enugh t respnd t changing needs. Using significant changes are nt pssible, since the cmpany culd indicate that the scpe changes impact ability t deliver. In sme cases, we may even identify alternate / additinal custmers. Test Centers stated: The end user makes all decisins n requirements. In general we require the end user t either be at all technical reviews r be the prject manager. Inputs and cmments cnfirm that these tasks are valid in the SBIR envirnment. SBIR requirements are captured at a tp level and tied t the research bjectives and d nt change typically within the shrt scpe f the phase. Requirements Management is perfrmed with increased rigr in later phases f the prject as it becmes mre relevant. Hwever Technlgy Directrates, Test Centers and ALCs shuld be accmplishing all f these tasks when ever requirements change. 41

55 Decisin Analysis Results frm the interviews identified that the tasks identified fr Decisin Analysis are nt always accmplished. Respnses were higher fr Technlgy Directrates in mst cases. Decisin Analysis Criteria selected fr decisin and methds t be used in cnducting the analysis. Analysis cnducted t help chse amng alternatives t achieve a balanced, supprtable, rbust, and cst effective prgram. Analysis methds include sme f the fllwing: trade studies, mdeling and simulatin, cst/benefit analysis, and the analytic hierarchy prcess (AHP). Studies are augmented with virtual and/r physical prttypes, where, prir t making decisins n best alternative. Ntewrthy r Significant Cmments Table 11: Decisin Analysis Technlgy Directrates Test Centers ALCs Ttal 64% 0% 33% 55% 64% 0% 33% 50% 50% 0% 100% 50% 50% 0% 67% 45% Technlgy Directrates stated: I lk at the perfrmance f the small business, the likelihd f transitin, the apprach, and fr innvatin. Identify the best apprach thrugh a multi-disciplinary team Feasible cst. Trade studies are typically nt accmplished. Like t use mdeling and simulatin when. This criteria business des nt track with SBIR award selectin criteria. We are bund by law t fllw the fairly vague, brad, and PUBLISHED criteria (e.g. technical merit, experience, dual-use/cmmercial/transitin ptential. The prjects themselves may embed technical trades, mdeling, feasibility demnstratin, prttype develpment. Respnses t the interview and additinal cmments varied fr Decisin Analysis. A limited amunt f Decisin Analysis is accmplished during Phase I and II 42

56 SBIR prjects due t the limited scpe. Fcus is n translating the research int slutins. SBIR prjects must wrk within the limited time frame and resurces allcated. Frmal analysis methds and prttypes are used nly when and the prject has the resurces. Technical Planning Technical Planning establishes and maintains dcumentatin that defines the technical aspects f the prject. Participants identified well the need t define the scpe f the effrt t include exit criteria, cnstraints and interfaces. Table 12: Technical Planning Technlgy Technical Planning Directrates Technical Planning made t ensure that the technical activities were cnducted prperly thrughut the system s life cycle. Define the scpe f the technical effrt required t achieve prgram technical gals which includes exit criteria and prducts/deliverables which can be tracked with prgress measured. Indentify cnstraints and interfaces that will result in derived technical requirements. Cntribute input t the Systems Engineering Plan, which is wned and maintained by the acquisitin activity. Ntewrthy r Significant Cmments 43 Test Centers ALCs Ttal 43% 0% 0% 30% 93% 100% 67% 90% 64% 100% 67% 70% 7% 100% 33% 20% Technlgy Directrates stated: I dn t knw what these mean. The planning ges as far as infrming the SBIR awardees the expectatins, why I like their innvative ideas, and any adjustments I think shuld be taken based n the apparent effectiveness f their prpsed wrk. Time tables are assciated with budget.

57 Reccurring interactin with cntractr is needed. IMP des nt apply fr prjects under 20M. Prir t prpsal fllwing tpic call RZ prvides feedback n expected evaluatin criteria. I am nt even sure what this means. We typically negtiate tpics with spnsrs, alng the lines f requirements. These tpics usually (but nt always) track with their tp pririties. It is subjective, because we and they are human. Smetimes as a result we live with nt the best defined tpics, and we have t d the best we can t ensure that the wrk is relevant. On the scale f the SBIR prgram, it is impssible t get this right 100% f the time. Test Centers stated: Fr AEDC, the end user is als the acquisitin activity. Our SBIR s are designed with a specific AEDC use in mind. We try t incrprate ther centers technical requirements t enable better cmmercializatin but these are secndary t the AEDC requirements. Technical Planning fr thrughut the life cycle and develpment f a Systems Engineering Plan (SEP) are tasks mre later in the develpment. Results fr thse areas identified them t nt be fr SBIR phase I and II prjects. Thugh a particular SEP may nt exist fr thse phases prject managers shuld identify and fllw the verarching SEP if ne exists fr the platfrm that the SBIR prject will eventually integrate t r the verarching rganizatinal SEP. Technical Assessment Several f the tasks fr Technical Assessment were identified during the interviews as displayed in green belw. Hwever tw f the tasks (highlighted in yellw) were identified as nt always. 44

58 Technical Assessment Measure technical prgress, technlgy maturity, and the effectiveness f plans and requirements. Activities include: Technical Perfrmance Measurement, Technlgy Readiness Assessment, and the cnduct f technical reviews. Table 13: Technical Assessment Demnstrate and cnfirm cmpletin f required accmplishments and S&T exit criteria. Discver deficiencies r anmalies that may result in the applicatin f crrective actin and may have frmed the technical prtin f a cntinuus prcess imprvement prcess when used t evaluate applicatin f SE IAW paragraph Technical assessment inputs used in supprt f the Labratry Management Review prcess. Technical assessment activities cnducted in cncert with existing reviews where pssible t minimize disruptin t the research prject. Ntewrthy r Significant Cmments Technlgy Directrates Test Centers ALCs Ttal 86% 100% 100% 85% 79% 100% 33% 70% 43% 100% 33% 45% 86% 0% 0% 71% 36% 0% 33% 35% Technlgy Directrates stated: As the cntract prgram manager, I d all f these n a subjective basis and a gut feeling using engineering judgment f best apprach. These things sund like they cme frm textbks r acquisitin training prgrams. Mst SBIRs are s shrt in duratin and fluid, nt t mentin limited in scpe, that sme f these activities (TRA) are t difficult t d n a recurring basis (like every 3-6 mnths). Als, it is nt uncmmn fr sme individuals t have mre than 12 SBIRs at any mment. They are usually a secndary duty, as we cannt affrd t dedicate even a single individual t 1 r 2 SBIRs. Test Centers stated: Every AEDC SBIR is required t have an nsite demnstratin at the midpint f the Phase II. We als encurage ur cntractrs t prvide a demnstratin at the end f Phase I in rder t shw a viable path t the required TRL. 45

59 The interview results and cmments identified the tasks during phase I and II SBIR prjects. The tw tasks that were identified in yellw may be in sme cases. Hwever due t the shrt duratin and limited scpe f the SBIR phase they may nt be. Overall Technical Assessment is a critical step in the technical management f the prject t ensure the prject is mature enugh t enter the next phase by meeting entry and exit criteria and shuld be assessed by all SBIR prject managers. Technical Data Management The SBIR cmmunity identified the frmal use f the Defense Technical Infrmatin Center. All frmal reprts are dcumented in DTIC. Table 14: Technical Data Management Technical Data Management Prject data managed thrugh the Defense Technical Infrmatin Center (DTIC) Or similar data base. Technlgy Directrates Test Centers ALCs Ttal 100% 100% 33% 85% 0% 0% 0% 0% Ntewrthy r Significant Cmments Technlgy Directrates stated: We have a fairly regular set f actins that are always perfrmed. We have kickff meeting, peridic technical interchange meetings and telecns, s n demand fr technical clarificatin, we review these in LMRs, we dcument research summaries and final technical reprts thrugh DTIC. We als respnd, n custmer demand, t participate in industry days r wrk with their wn database initiatives, which cme and g ver time. Air Lgistics Centers stated: Final reprts are put in DTIC, we use ur lcal server t stre all cntract activities. 46

60 Risk Management A gd risk management strategy is critical fr a successful prject. The large majrity f participants identified that the five risk management steps apply t the SBIR cmmunity. 65% identified acknwledged the existence f a risk management plan fr their prject. Hwever feedback frm the interviews identified that the Small Business perfrm mst aspects f the risk management. Table 15: Risk Management Risk Management Technlgy Directrates Test Centers ALCs Ttal Develp risk management plan and perfrmed 57% 100% 100% 65% the fllwing: Identified risk 86% 100% 100% 85% Analyze risk and define prbably and likelihd. 64% 100% 100% 70% Identify handling ptins 64% 100% 100% 70% Mitigate risk 79% 100% 100% 80% Track risk 86% 100% 100% 85% Ntewrthy r Significant Cmments Technlgy Directrates stated: Risk is tracked by the prject fficer. Infrmatin is prvided by cntractr (varies n hw ften and well PO interacts and fllws the cntractrs prgress). AFPAM is nt really being used as a guide frmally. Limited knwledge f it in wrk envirnment. In phase 1s, there is very little time t d any f this except in the brief interchanges that ne can have ver a 6-9mnth technical activity. They either get it r they dn t, in which case yu request / dn t request a Phase 2 prpsal. This is Darwinian. Yu then hpe that the best nes will be awarded, but it is a cmpetitive prcess, s smetimes even a perfect Phase 1 ends withut a fllw-n. The steps yu utline abve are typically dne in a very seat f pants way, but they typically ARE dne. Risk in a SBIR is generally limited t technical risk as cst is fixed and schedule deliverables are detailed. 47

61 Test Centers stated: Occasinally use canned Risk Analysis sftware (ie Crystal Ball). We understand every SBIR carries prgrammatic risk. Therefre these activities are nly dne fr Phase III s. Fr SBIR, we mainly use canned risk analysis sftware and ccasinally use Mnte-Carl methds t cnduct risk trade analyses. Air Lgistics Centers stated: SBIR prgrams are inherently at a high technical risk due t the nature f new technlgical develpment. Hwever, risk is managed by the utline abve. I am always trying t identify schedule, cst and technical risk early and wrking with the cntractr t mitigate. Early detectin is essential t risk management and verall prgram success. Risks n ur prjects are what we ver cme in the SBIRS the final analysis and subsequence prttype testing will validate success. Respnses and cmments identified that the SBIR cmmunity is aware f risk management techniques and they are t SBIR prjects. Prject managers in many cases rely n infrmatin frm the cntractr fr their assessments. Prject managers must ensure they fllw up n a regular basis with the cntractr t check the health f the prject and ensure the risk is being managed effectively and the prject is n scpe. Cnfiguratin Management Mst participants identified the need t dcument results fr Cnfiguratin Management. Hwever there were mixed results frm the interview respnses. 48

62 Table 16: Cnfiguratin Management Technlgy Cnfiguratin Management Directrates Ensure the repeatability f experimental results t include data by knwing and keeping a recrd f the labratry set-up as well as tracking changes t it. Keep a recrd f labratry experimental hardware cnfiguratin when measurements are gathered including such things as calibratin status, envirnmental cnditins, sftware versin and mdificatins used, and dcumentatin (data) resulting frm the experiment/demnstratin? A cmplete audit trail f decisins affecting labratry equipment/sftware design mdificatins maintained. Ntewrthy r Significant Cmments Test Centers ALCs Ttal 64% 50% 33% 55% 43% 50% 33% 40% 29% 0% 0% 20% Technlgy Directrates stated: Nne, these d nt seem t apply t the type f SBIR s I manage. Seems mre inhuse research as ppsed t SBIR cntracts in which are perfrmed by external t WPAFB cntractrs. Cntractr maintains CM recrds. PO des nt have a CMP. Gv t PO is nt very invlved with ensuring CM is accmplished n average. Final technical reprt is published, t dcument permanently what we did d in terms f the things yu describe abve. Cnfiguratins fr fficial tests and dems are dcumented, we als rely n cntractr supplied ICDs, schematics, etc. Test Centers stated: AEDC nly cnfigures nce the SBIR is ready fr transitin t an actual test cell. This nrmally happens at least ne year fllwing the clse-ut f the phase II. Fr Phase III acquisitins, AEDC fllws regular SE prcesses and requires full cnfiguratin management as part f the acquisitin. Air Lgistics Centers stated: Perfrmed by the SBIR cntractr, the infrmatin is cntained in their interim reprts, als much f the testing is perfrmed by an independent testing facility, which will identify the equipment used and calibratins. These data sheets are included with the reprts. 49

63 Results and cmments are cncerning since it seems that Cnfiguratin Management is dependent n the cntractr and nt well regulated by the SBIR prject managers. A fundamental basic f gd SE is t maintain gd cnfiguratin management fr the life cycle f the system. S&T prjects are derived t supprt a ptential capability need r requirement. Thus all prject results shuld be dcumented t ensure traceability t the higher level requirements as well repeatability f results. Lack f prper dcumentatin and traceability risks wasting effrts that d nt supprt the prject bjectives as well as increase the challenges f transitining t the next phase with well dcumented repeatable results. Interface Management Similarly t Cnfiguratin Management respnses varied and nt many Interface Management tasks were nt validated as being. Table 17: Interface Management Technlgy Interface Management Directrates Ensure interface definitin and cmpliance amng the elements that cmpse the labratry system (internal interfaces), as well as with ther systems with which the peratinal system r system elements might interact (external interfaces). Ensure that all internal and external interface requirement changes are prperly dcumented in accrdance with the cnfiguratin management plan and cmmunicated t all affected elements f the prgram. All interfaces defined in sufficient detail t facilitate necessary cmmunicatin/interactin amng system, subsystem, and cmpnents. 50 Test Centers ALCs Ttal 50% 0% 0% 40% 50% 0% 0% 35% 71% 0% 33% 55%

64 Ntewrthy r Significant Cmments Technlgy Directrates stated: Nne apply t the type f SBIR s I manage. My SBIR s relate t cmpnent maturatin rather than system engineering. We are trying t demnstrate a higher perfrmance cmpnent where we need t be aware f the mating technlgies. Hwever, interfacing usually cmes much beynd Phase II and ften times beynd Phase III cntracts. Defined fr Demnstratin cnfiguratin in shwing prgram met the tpic gals. Cntractr maintains IM recrds. There shuld be stake hlder invlvement t ensure interface interactins are defined and acceptable. Nthing this frmal, except in rare ccasins. Test Centers stated: AEDC des nt have SBIRs that are intended t be inserted int ther systems and therefre d nt require ICD s. When we transitin a SBIR all f the abve ccurs but nt during a SBIR. Air Lgistics Centers stated: We wrk with structural type systems (Shelters, radmes and twers). Each is unique t the individual missin. We dn t have frmal external interfaces. We d have industry standards and lcal cdes fr gegraphic lcatin that they must meet and pass. A lw level f rigr fr Interface Management may nly be required fr many typical SBIR prjects because nt all interfaces may yet be fully defined. Hwever stakehlder invlvement and early architecture effrts shuld identify the interfaces necessary and drive the requirements fr early Interface Management. Overall the SBIR cmmunity did nt identify well with the tasks identified. Evidence suggest that SBIR prject managers really heavily n the Small Business t manage interfaces. This is a ptential risk since gd interface management is required t be successful in transitining the prject. 51

65 Architectural Design Respnses fr Architectural Design did nt validate any f the tasks. Table 18: Architectural Design Technlgy Technlgy Directrates Directrates Translate the utputs f the Stakehlder Requirements Definitin and Requirements Analysis prcesses int alternative technical slutins and selects a final technical path t explre. Iterate Stakehlder Requirements Definitin, Requirements Analysis, and with the technical management prcesses t identify and select the best slutin by first develping a high-level view f the system architecture capable f meeting stakehlder needs. Output the design functinal r physical architecture sufficiently detailed t allw upward and dwnward traceability f requirements. Ntewrthy r Significant Cmments Technlgy Directrates stated: Test Centers ALCs Ttal 29% 0% 0% 20% 29% 0% 0% 25% 21% 0% 33% 20% Architectural design is nt accunted. Rather, we design t meet an bjective f a cmpnent perfrmance. Dn t feel architectures apply much fr phase I and II prjects. Architectural design is nt accunted. Rather, we design t meet an bjective f a cmpnent perfrmance. Frmal Architectural Design rarely has use in the wrld f SBIR because the custmers aren t cncerned abut this srt f frmal representatin f impact frm a SBIR effrt. I am sure we wuld d it if we sensed any utility. Air Lgistics Centers stated: In ur case, the stakehlder cmes int play after we ensure the prduct meets the basic standards such as ANSI 1925 fr shelters. Then the stakehlders cme int play t incrprate their system int ur prduct. 52

66 Respnses and cmments fr Architectural Design are very cncerning. S&T prjects risk being ut f scpe with DD bjectives withut maintaining traceability t requirements. Early architectural mapping t thse requirements ensure that prjects stay n scpe. Results identified that the majrity f participants d nt frmally dcument architecture and many d nt feel architecture is t SBIR which is a discuraging miscnceptin. Even Basic Research shuld map t a high level capability. Implementatin Capturing the right infrmatin t address the preparatin required t supprt the prject transitin t the next phase n aspects f prductin f prducts and/r services is an imprtant part f the prject. The majrity f participants identified with the belw tasks. Hwever, as results shwed belw, all are tasks in the SBIR cmmunity but nly 50 and 55% identified with tw f the task. Table 19: Implementatin Implementatin Yield the fundamental capability f the prgram. Include sme testing f the individual elements befre they passed t Integratin. Develp supprting dcumentatin fr the system; such as the as-built cnfiguratin, r discvered limitatins f the cncept; is als a part f the implementatin prcess. Technlgy Directrates Test Centers ALCs Ttal 64% 50% 33% 55% 79% 100% 67% 75% 43% 100% 67% 50% 53

67 Ntewrthy r Significant Cmments Technlgy Directrates stated: We ensure we are meeting sme srt f perfrmance requirement befre attempting t lk at integratin r mating with ther technlgies. Hwever, this usually cmes at the end f a SBIR prgram where there are nt usually sufficient funds left fr maturing any mre. Include sme testing f the individual elements befre they passed t integratin We test specific assertins that cmprise the prgram statement f wrk. T the degree they embdy these things, we d them. They are certainly dcumented variusly thrugh the status reprts and final technical reprts that are always required. Test Centers stated: AEDC will nrmally accept the deliverable and test it in a lab quite sme time prir t transitin t an actual test cell. Air Lgistics Centers stated: Update T.O.s & Drawings, publish new ASTM, etc., Standards. Fr Jint-Service Teams t implement t wider DD activities. Implementatin is accmplished thrugh capturing all technical requirements within an Air Frce specificatin which then rlls up int Technical Manuals (T.O. s). Results and cmments verall were psitive fr Implementatin. Mst f the SBIR cmmunity interviewed understd the imprtance f Implementatin. It hwever accmplished with limited rigr during SBIR phase I and II due t the limited scpe and early develpment effrt. Implementatin shuld increase in rigr as the prject matures. Integratin Results fr the Integratin tasks identified that the tasks are typically nt in the SBIR phase I and II envirnment. 54

68 Integratin Incrprate the lwer-level system elements int a higher-level system element in the physical architecture. Define the plan r strategy fr the Integratin prcess, including the assembly sequence, that may have impsed cnstraints n the design slutin Ntewrthy r Significant Cmments Table 20: Integratin Technlgy Directrates Test Centers ALCs Ttal 36% 50% 100% 45% 43% 50% 67% 45% Technlgy Directrates stated: We wrk with the prime cntractrs t see if there is interest. If there is, we lk fr funding t integrate the technlgy int sme relevant system. Unfrtunately, things usually die early due t insufficient funds. Integratin as need t demnstrate the SBIR gals nt a frmal plan but may be art f the test plan. D nt dictate the prcess, nly indicate the requirements. Wrk with the cntactr t see if there is interest t integrate a SBIR int a relevant system and try t lcate funding. Very limited in Phase I and II SBIRs. Air Lgistics Centers stated: Generally we take ther Systems and incrprate them in ur shelter, put n a twer r cver them up with a radme. As results and cmments identified a larger emphasis n integratin takes place in Phase III f a SBIR prject. Thus results fr the tasks identified frm AFRL plicy fr integratin were nt identified as by the majrity f participants fr the Phase I & II SBIR cmmunity. These tasks may r may nt be depending n the maturity f the prject. ALCs did identify with these tasks likely because they have mre initial infrmatin n the platfrm the SBIR prject will be integrated t supprt. 55

69 Verificatin Respnses identified that SBIR prjects verify sme elements f the prject. Hwever, the tasks in yellw were identified as nt fr SBIR phase I and II prjects. Table 21: Verificatin Technlgy Verificatin Directrates Cnfirm that the labratry/experimental system element meets design specificatins. Test the system elements against their defined requirements (predicted versus experimental results). Design slutins at all levels f the physical architecture were verified thrugh a csteffective cmbinatin f analysis, examinatin, demnstratin, and testing, all f which can be aided by mdeling and simulatin. Answer the verificatin questin Did we build the thing right?. Ntewrthy r Significant Cmments 56 Test Centers ALCs Ttal 86% 50% 100% 80% 64% 50% 100% 65% 29% 0% 67% 30% 29% 50% 67% 40% Technlgy Directrates stated: Sme verificatin takes place with cmpnent testing. Dems are usually jint test events and we try t test the capability against peratinal expectatins. Air Lgistics Centers stated: Perfrm Live-Fire, and shp level perfrmance and testing (Zinc-Nickel plating t replace Cadmium n Landing Gear cmpnents). It s mre subtle and cmplex than these simple menu chices. Results and cmments identified that SBIR phase I and II prjects nly accmplish a limited amunt f Verificatin testing. Further verificatin testing will be accmplished in later phases. Thus answering the verificatin questin Did we build the

70 thing right? will be accmplished later in the develpment f the prject. Similarly design slutins fr all levels f the physical architecture are mre in the later phases f the prject. Validatin Respnses identified Validatin tasks are accmplished fr SBIR prjects. Hwever the validatin questin Did we build the right thing was identified as nt always. Validatin Test the perfrmance f the technlgy against the riginal prgram gals. Capture any testing results/data s that they are available fr further develpment/research/maturatin effrts. Answer the validatin questin Did we build the right thing?. Ntewrthy r Significant Cmments Table 22: Validatin Technlgy Directrates Test Centers ALCs Ttal 86% 100% 67% 80% 79% 100% 67% 75% 29% 50% 100% 45% Test Centers stated: It is rare in a SBIR develpment that we receive exactly what we want. Clearly, the small business is interested in cmmercializatin rather than simply delivery f the prttype. Because f this, AEDC must ften spend additinal missin resurces t ensure the system meets ur requirements. We d this thrugh additinal develpment nce the prttype has been received. Usually nt enugh funding thrugh a SBIR fr validatin. Full validatin takes place as a SBIR is integrated int a system thrugh ther funding methds. Smetimes, n the scale f a SBIR, we dn t even get a WHOLE thing, and we smetimes cannt answer these simplistic questins like did we build the right thing All are dne infrmally. 57

71 SBIR Phase I and II prjects nly accmplish a limited amunt f Validatin testing due t their limited scpe and duratin. Further validatin testing will be accmplished in later phases. Transitin Mst participants identified the imprtance f cnsidering and applying steps fr their prjects t transitin t the next phase. Table 23: Transitin Technlgy Transitin Directrates Deliver a supprtable technlgy prject capable f being put in the hands f the warfighter. The transitin prcess applied in a step-bystep manner t mve the technlgy t the next level in the develpmental cycle. Needs f fllw-n phases cnsidered early in the prgram and included in all f the technical management prcesses. Ntewrthy r Significant Cmments 58 Test Centers ALCs Ttal 43% 50% 67% 45% 86% 50% 67% 75% 71% 50% 100% 75% Technlgy Directrates stated: Nt much cnsideratin during phase I and phase II. There is n such thing as a step by step transitin prcess. This seems t reflect a fairly meager understanding f technlgy develpment. Yu are nt always able t mature even a piece f a prblem t a level that can be transitined. Usually yu target a large defense cntractr/spo/gv agency that can integrate the tech int their prducts rather than straight t warfighter/prductin. Air Lgistics Centers stated: In phase II we fcus n ne system r custmer that has a need fr the technlgy. We find that new technlgy is met with resistance. That it is imprtant t have a wrking prttype t custmers can kick the tires r see a physical item.

72 As results and cmments validated Transitin tasks fr SBIR nt all tasks were identified as. Deliver a supprtable technlgy prject capable f being put in the hands f the warfighter is an actin that takes place in Phase III and is nt directly t phase I & II prjects until they prgress t that phase. That is why nly 45% f the participants identified with that task. Sme SBIR prjects never reach phase III. Summary f Results Results belw include all the participants. It illustrates that hw SBIR is unique since fr a majr acquisitin prgram all tasks wuld be. Figure 7: All Participant Results 59

73 Center Results Results identified several weak areas with regards f implementatin f AFRL SE tasks frm identified in AFRL plicy. This illustrates hw SBIR is unique when cmpared t typical S & T prjects within AFRL. Figure 8: Technlgy Directrates Results 60

74 Test Center Results Since the interview instrument was develped using AFRL plicy sme f the tasks were in AFRL language and participants did nt identify with them. Data was gathered frm a small participant size fr Test Centers. Bth f thse factrs must be cnsidered when intrepreting results. Additinal cmments n hw tasks were actually perfrmed was captured in their cmments that was previusly nted fr each area. Architectural Design, Decisin Analysis and Interface Management identified n interview SE tasks as frm survey respnses. Figure 9: Test Centers Results 61

75 Air Lgistics Center Results Since the interview instrument was develped using AFRL plicy sme f the tasks were in AFRL language and participants did nt identify with them. Data was gathered frm a small participant size fr ALC s. Bth f thse factrs must be cnsidered when intrepreting results. Additinal cmments n hw tasks were actually perfrmed was captured in their cmments that was previusly nted fr each area. Figure 10: Air Lgistics Centers Results 62

76 AFRL Eight Systems Engineering Key Questins AFRL emphasizes the use f the Eight SE Key Questins identified in AFRLI The questins are mapped back t the DAG SE prcesses. This belw table illustrates the interview results mapped back t the questins. The majrity f interview participants identified these questins t be useful t extremely useful. Each participant subjectively identified the mst and least useful questin illustrated in the belw table. Table 24: AFRL SE Key Questins Mapped t Interview Results Mst useful Least Useful 1. Wh is yur custmer? What are the custmer s requirements? 5 3. Hw will yu demnstrate yu have met the requirements? What are the technlgy ptins? Which is the best apprach? 1 6. What are the risks t develping the selected technlgy? Hw will yu structure yur prgram t meet requirements and mitigate risk? 2 8. What is yur business-based transitin plan that meets custmer apprval? 4 Mapped t SE Prcess Requirements Management Requirements Definitin Requirements Management Cnfiguratin, Data & Interface Management Requirements Develpment & Validatin Technical Planning & Technical Assessment Decisin Analysis Risk & Requirements Management Verificatin, Validatin & Transitin Cnfiguratin & Interface Management Integratin Lgical Analysis Technical Planning & Implementatin Lgical Analysis Technical Assessment Decisin Analysis & Implementatin Integratin Design Slutin Technical Planning Risk Management Design Slutin Risk Management & Data Management Technical Planning & Requirements Definitin Verificatin &Validatin Implementatin & Transitin Lgical Analysis & Design Slutin Integratin Cnfiguratin Management Data Management & Interface Management Transitin 63

77 Thugh the majrity f participant thught the questins were useful abut 30% identified them as nt being that useful. Several participants bserved if yu can t answer these yu are nt managing the prject crrectly. The first three seemed t be the mst imprtant. The least imprtant questin was identified as questin # 8. This is likely because nt all SBIR prjects transitin int larger prjects. The analysis f hw these questins map back t the DAG prcesses suggests these questins must be answered t successfully manage a prgram. Hwever different rganizatins manage SBIR prjects which value and implement SE differently. The difference in pinin frm participants n their value addresses a bigger issue f the different levels f understanding and appreciatin f SE. In the authr s experience tw majr factrs are respnsible. Either wrkers have nt received gd SE educatin and have nt made the cnnectin that gd SE leads t gd prgram management r they have been discuraged frm their experience with mismanaged SE effrts. SE must be tailred t the apprpriate level fr a prgram r prject t ensure it is value added. Histrically the DD had tried t standardize SE prcesses by mandating them. This ne size fits all apprach ften leads t an increased wrk lad withut much value added t the prgram. One participant identified that the first seven questins are almst ffensive since they must be knwn. The authr is nt surprised hw an experienced prject manager can see it this way hwever the management f SBIR prjects varies frm prject managers new t the gvernment t the very seasned. This pint is evident in the results abve and the varying understanding f the questins. 64

78 AF SEAM Cmparisn The authr cmpared his results with AF SEAM t identify the SE tasks fr the SBIR cmmunity. AF SEAM and AFRL plicy d nt directly align since AFRL plicy maps back t the DAG. Using the data gathered frm this research the authr reviewed each AF SEAM task with the data cllected and translated thse results int AF SEAM SE tasks. Results f this cmparisn are captured in Appendix 2. This infrmatin will be very useful t the Air Frce SBIR cmmunity when the current draft revisin f is published since it is prjected t align with Air Frce SEAM. It can be used t explain what is t the SBIR cmmunity frm AF SEAM and als illustrates that AF SEAM is nt tailred specific fr the SBIR cmmunity. Only abut 50% f the tasks frm AF SEAM were fund t be fr the SBIR cmmunity. Many f the tasks identified in AF SEAM are nt until later phases f a prgram. These findings shw that it wuld nt be useful t implement AF SEAM within the SBIR cmmunity. In additin t nly half the tasks being, AF SEAM requires a large manpwer effrt t cmplete it due t its 190 SE tasks. Since SBIR prjects d nt have resurces t supprt such a significant SE effrt and it is nt tailred fr SBIR a mre tailred apprach is wuld be a much better use f resurces. 65

79 Research Questins Answered 1. Hw well d SBIR prjects currently implement Systems Engineering plicy? Answer: The SBIR cmmunity des nt believe all SE tasks in AFRL plicy apply t their prjects and des cnsistently implement SE tasks. 2. Hw are SE Plicies implemented? Answer: The results identified a wide spectrum f interpretatin that partially rests n the spnsring rganizatin type and the SBIR phase. Thse results identified weak areas within the current plicy. 3. Hw d DD and Air Frce SE prcesses apply t SBIR prjects? Answer: DD and Air Frce SE prcesses d apply t SBIR prjects. Hwever they must be tailred fr the scpe f the prject. Applicable SE tasks SBIR prjects were identified in Chapter IV. 4. T what level f rigr des each SE prcess apply t SBIR prjects? Answer: The number f tasks fr each SE prcess area varies. Specific tasks fr each area where captured in interview results and cmparisn f AF SEAM tasks. 66

80 5. What is the best way t implement these prcesses? Answer: The best way t implement SE t a SBIR prject is establish a tailred apprach and fllw the key steps utlined in plicy. SE guidance fr this is identified in the AFRL SE Guide. Prject managers must have a gd understanding f SE and tailr a slid apprach fr their prject. Prject fficers can use results frm this research as a guide t better understand what level f rigr typically applies t a SBIR prject. Summary Results frm the literature review indentified a number f SE prcesses that were applied effectively t AFRL prjects. The interview data identified what SE tasks are being accmplished and t what level f rigr within the SBIR cmmunity. The results identified in many areas that the SE tasks defined in plicy are either nt r are nt being accmplished within the SBIR cmmunity. Better SE educatin will help prject managers fully tailr SE t their prjects and ensure tasks are being incrprated. Tasks that are nt hwever shuld nt be required. The results frm the literature reviews and interviews identified thse tasks and can be used t better tailr an rganizatin s SE apprach fr SBIR prjects t avid wasting resurces n nn value added tasks. Overall these findings again illustrated the unique nature f SBIR prjects. 67

81 V. Cnclusins and Recmmendatins Chapter IV identified the SE tasks that are t the SBIR cmmunity. The case studies demnstrated successfully tailring a SE apprach fr S&T prjects. Interview results prvided SE tasks frm AFRL plicy fr SBIR prjects. The authr translated thse results with his SE expertise t identify what SE tasks are fr SBIR frm Air Frce SEAM in Appendix 2. Analysis f AFRL SE tasks and AF SEAM identified neither f them are specific enugh fr SBIR. Many f the tasks were nt during SBIR phase I and II prjects due t the unique nature f SBIR which includes limited budget, shrt schedule and a limited scpe. Prjects als vary greatly acrss the Air Frce and DD. This study cncludes that current plicy des nt fully define SE fr the SBIR cmmunity and that SE is being implemented at varius level amngst the different rganizatins that manage SBIR prjects. Results frm this study als identified that verall the SBIR cmmunity was well educated and understd hw certain SE prcesses applied t their prjects. Hwever the results als identified that they are very week in many areas. I believe there is a miscnceptin within the cmmunity that sme areas f SE d nt apply t their prjects. All areas f SE apply with different levels f rigr fr any prject. Leadership and prject managers must ensure adequate levels f SE are being incrprated int their prjects t imprve their chance f success, limit cst and schedule verruns and meet perfrmance gals. Failure t fllw established SE prcesses in any ne area can have significant negative cnsequences t the prject. 68

82 SBIR SE Checklist The fllwing checklist will ensure adequate levels f SE are being incrprated fr SBIR prjects. This SBIR SE Checklist is a guide fr prject managers and engineers t ensure all SE areas are adequately addressed. This checklist was develped using the results derived frm this research and align with the 10 AF SEAM SE prcesses areas. Represents general AF SE prcess tasks tailred fr SBIR - Represents specific SE tasks captured in analysis Table 25: SBIR SE Checklist Requirements Prject Planning Risk Management Decisin Analysis Design Determine requirements t include stakehlder needs, expectatins, cnstraints, and interface requirements. - Translate all stakehlder needs t technical requirements. - Requirements made quantifiable, have unique definitins, and specified threshlds and bjectives. - Wrk with stakehlders t refine requirements. - Perfrmance parameters and cnstraints allcated and derived technical requirements defined. - Maintain the traceability f all requirements frm needs. - Dcument changes and recrd ratinale f changes. Identify prject milestnes t include cst, schedule and technical milestnes. - Define the scpe f the tech effrt required t achieve prgram technical gals. - Define exit criteria and prducts/deliverables which can be tracked with prgress measured. Develp a risk management plan and identify, analyze, identify handling ptins, mitigate and track risk. Establish selectin criteria, identify & evaluate alternatives and select slutin. - Criteria selected fr decisin & methds t be used in cnducting the analysis. - Identify analysis methds and cnduct analysis f alternatives. Establish the design and integratin baseline. - Incrprate the lwer-level system elements int a higher-level system element in the physical architecture. - Indentify cnstraints & interfaces that will result in derived technical requirements. 69

83 Technical Management & Cntrl Cnfiguratin Management Verificatin & Validatin Transitin, Fielding, & Sustainment Manufacturing Establish and maintain the prject envirnment, integrated prduct teams (IPT), measurements apprach and mnitr technical reviews, wrk prducts, prject data, crrective actins and technical milestnes. - Measure technical prgress, technlgy maturity and the effectiveness f plans and requirements. - Demnstrate and cnfirm cmpletin f required accmplishments and prject exit criteria. Establish the technical baseline, track and dcument changes. - Maintain recrd f all cnfiguratins t include hardware, sftware and test set up and dcument changes. - Define internal and external interfaces. - Prject data managed thrugh the Defense Technical Infrmatin Center (DTIC). Establish and maintain the verall verificatin strategy and plan t include verificatin and validatin criteria and an integrated testing apprach when. Verify and Validate that the prject has meets the required parameters. - Cnfirm prject meets design specificatins. - Test the system elements against their defined requirements. - Test the perfrmance f the technlgy against the riginal prgram gals. Indentify future transitin, fielding, & sustainment requirements as needed t prceed t the next phase f the prject. - Needs f fllw-n phases cnsidered early in the prgram and included in all f the technical management prcesses. - Yield the fundamental capability f the prgram. Identify and maintain dcumentatin relevant t the future prductin f the prject. - Develp supprting dcumentatin fr the system. Prject Managers using this checklist will begin t accmplish sme f these tasks in Phase 1 with the emphasis f demnstrating the prject is feasible, identifying stakehlders and defining requirements. By the end f Phase 2 all f the abve tasks shuld have been tailred and accmplished fr the prject. Prjects that enter Phase 2.5 will emphasize n further defining and dcumenting infrmatin and demnstrating the technlgy with the hpes t aid in the future transitin f the prject t the Phase III. Phase III is the cmmercializatin phase. 70

84 Significance f Research This research identified the SE plicy gap in the SBIR cmmunity and defined the SE tasks. This highlights a huge risk as millins f dllars are spent within the DD each year n SBIR prjects. Failure t implement gd SE principles can and will lead t cst verruns, schedule slips and perfrmance shrt falls. Findings frm this research shuld be used t tailr a SE apprach fr SBIR prjects t ensure SE practices are being implemented in a best practice manner. Recmmendatins fr Actin 1. Organizatinal plicy needs t be tailred fr SBIR. The SBIR cmmunity shuld use the identified SBIR SE tasks frm this study t develp adequate plicy and SE tasks fr their SBIR prjects. SE experience varies greatly amngst the SBIR prject manager within the DD. Yu may have a prject manager with 30 years f experience r a newly cmmissined fficer managing the prject. There fr it is critical t have adequate SE plicy and guidance in place t ensure that critical tasks are being accmplished. 2. The SBIR cmmunity shuld incrprate a tailred SE apprach fr their prjects. The apprach shuld be cnsistent with the Streamline SE Apprach fr the S&T cmmunity utlined in the draft AFRL SE Guide. The case study review fr this research identified the benefits f using such an apprach. 71

85 3. The SBIR cmmunity shuld ensure the prject managers receive adequate SE educatin t enable them t tailr SE t their prjects. As the scpe f SBIR prjects can vary greatly it can be challenging fr prject managers t understand hw all areas f SE apply t their prjects. Results frm the interviews als identified weak areas in the cmmunity as well as miscnceptins that sme areas dn t apply t them. That is why gd SE educatin is critical fr prject managers t truly make the cnnectin f hw SE applies t their prjects. The Air Frce and DD have many gd resurces t prvide SE educatin t the wrk frce such as DAU and AFIT. Supervisrs shuld ensure they are requiring their flks t take advantage f these pprtunities and are cntinuing t develp their prject manager skills. Recmmendatin fr Future Research The research reveled several pprtunities fr future wrk that was nt within the scpe f the research. 1. Gd SE practices are ften hard t measure. Further data culd be cllected frm each f the SBIR rganizatin managing prjects t identify the successful transitin rate f their prjects and the SE rigr being implemented within the rganizatins. This wuld likely illustrate the impact f implementing gd SE practices int the SBIR cmmunity. Transitin data was nt available fr this research prject. 72

86 Appendix 1 SBIR SE Interview Small Business Innvative Research (SBIR) Technical Management Prcesses Interview Date: Survey Directins: Please answer each questin in regards t the SBIR prjects yu have managed. Experience Questins 1. What is yur current jb title? (Circle ne) Prgram Manager Prject Engineer. Chief Engineer Other 2. What AFRL Directrate d yu wrk fr? (Circle ne) RX RY RZ RB RH RD RV RW 711 HPW AFOSR Other 3. Hw are yu emplyed? (Circle ne) Military Civilian Cntractr Other 4. Hw many years f experience d yu have in yur jb? (Circle ne) What level f APDP certificatin have yu accmplished? (Circle ne) nne 6. In what APDP area? (Circle ne) PM SPRDE Other 7. What phase f SBIR prjects and hw many have yu managed? Phase I Phase II Other 73

87 8. What kinds f Stakehlders Requirements Definitin d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: All inputs frm relevant stakehlders translated int technical requirements. Requirements made quantifiable, have unique definitins, and specified threshlds and bjectives. Wrk with the user t establish and refine gals, attributes, perfrmance parameters, and financial and schedule cnstraints, and then ensure that all relevant requirements are addressed during the science and technlgy effrt. Translate the custmer needs int S&T prgram and system requirements. Other 9. What kinds f Requirements Analysis d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Obtain sets f lgical slutins t imprve understanding f the defined requirements and the relatinships amng the requirements (e.g. functinal, behaviral, tempral). Perfrmance parameters and cnstraints allcated and derived technical requirements defined. Partitin the technical prblem int self-cntained, chesive, lgical grupings f elements and, where apprpriate, defined the key interfaces. Other 10. What kinds f Requirements Management d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Maintain the traceability f all requirements frm needs Dcument all changes t thse requirements Recrd the ratinale fr thse changes. Traceable t sme current r ptential future military capability need. Other 11. What kinds f Decisin Analysis d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Criteria selected fr decisin and methds t be used in cnducting the analysis. Analysis cnducted t help chse amng alternatives t achieve a balanced, supprtable, rbust, and cst effective prgram. Analysis methds include sme f the fllwing: trade studies, mdeling and simulatin, cst/benefit analysis, and the analytic hierarchy prcess (AHP). Studies are augmented with virtual and/r physical prttypes, where, prir t making decisins n best alternative. Other 74

88 12. What kinds f Technical Planning d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Technical Planning made t ensure that the technical activities were cnducted prperly thrughut the system s life cycle. Define the scpe f the technical effrt required t achieve prgram technical gals which includes exit criteria and prducts/deliverables which can be tracked with prgress measured. Indentify cnstraints and interfaces that will result in derived technical requirements. Cntribute input t the Systems Engineering Plan, which is wned and maintained by the acquisitin activity. Other 13. What kinds f Technical Assessment d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Measure technical prgress, technlgy maturity, and the effectiveness f plans and requirements. Activities include: Technical Perfrmance Measurement, Technlgy Readiness Assessment, and the cnduct f technical reviews. Demnstrate and cnfirm cmpletin f required accmplishments and S&T exit criteria. Discver deficiencies r anmalies that may result in the applicatin f crrective actin and may have frmed the technical prtin f a cntinuus prcess imprvement prcess when used t evaluate applicatin f SE IAW paragraph Technical assessment inputs used in supprt f the Labratry Management Review prcess. Technical assessment activities cnducted in cncert with existing reviews where pssible t minimize disruptin t the research prject. Other 14. What kinds f Technical Data Management d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Prject data managed thrugh the Defense Technical Infrmatin Center (DTIC) Or similar data base. Other 75

89 15. What kinds f Risk Management d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Develp risk management plan and perfrmed the fllwing: Identified risk Analyze risk and define prbably and likelihd. Identify handling ptins Mitigate risk Track risk 16. What kinds f Cnfiguratin Management d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Ensure the repeatability f experimental results t include data by knwing and keeping a recrd f the labratry set-up as well as tracking changes t it. Keep a recrd f labratry experimental hardware cnfiguratin when measurements are gathered including such things as calibratin status, envirnmental cnditins, sftware versin and mdificatins used, and dcumentatin (data) resulting frm the experiment/demnstratin? A cmplete audit trail f decisins affecting labratry equipment/sftware design mdificatins maintained. Other 17. What kinds f Interface Management d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Ensure interface definitin and cmpliance amng the elements that cmpse the labratry system (internal interfaces), as well as with ther systems with which the peratinal system r system elements might interact (external interfaces). Ensure that all internal and external interface requirement changes are prperly dcumented in accrdance with the cnfiguratin management plan and cmmunicated t all affected elements f the prgram. All interfaces defined in sufficient detail t facilitate necessary cmmunicatin/interactin amng system, subsystem, and cmpnents. Other 76

90 18. What kinds f Architectural Design d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Translate the utputs f the Stakehlder Requirements Definitin and Requirements Analysis prcesses int alternative technical slutins and selects a final technical path t explre. Iterate Stakehlder Requirements Definitin, Requirements Analysis, and with the technical management prcesses t identify and select the best slutin by first develping a high-level view f the system architecture capable f meeting stakehlder needs. Output the design functinal r physical architecture sufficiently detailed t allw upward and dwnward traceability f requirements. Generate sme f the fllwing: AV-1, SV-1, OV-1 r additinal DODAF 2.0 views Other 19. What kinds f Implementatin d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Yield the fundamental capability f the prgram. Include sme testing f the individual elements befre they passed t Integratin. Develp supprting dcumentatin fr the system; such as the as-built cnfiguratin, r discvered limitatins f the cncept; is als a part f the implementatin prcess. Other 20. What kinds f Integratin d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Incrprate the lwer-level system elements int a higher-level system element in the physical architecture. Define the plan r strategy fr the Integratin prcess, including the assembly sequence, that may have impsed cnstraints n the design slutin Other 21. What kinds f Verificatin d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Cnfirm that the labratry/experimental system element meets design specificatins. Test the system elements against their defined requirements (predicted versus experimental results). Design slutins at all levels f the physical architecture were verified thrugh a csteffective cmbinatin f analysis, examinatin, demnstratin, and testing, all f which can be aided by mdeling and simulatin. Answer the verificatin questin Did we build the thing right?. Other 77

91 22. What kinds f Validatin d yu accmplish n average fr yur SBIR prjects? Check all that are accmplished: Test the perfrmance f the technlgy against the riginal prgram gals. Capture any testing results/data s that they are available fr further develpment/research/maturatin effrts. Answer the validatin questin Did we build the right thing?. Other 23. What steps d yu take, r tasks d yu perfrm, t help with the pssible future Transitin f yur SBIR prjects? Check all that are accmplished: Deliver a supprtable technlgy prject capable f being put in the hands f the warfighter. The transitin prcess applied in a step-by-step manner t mve the technlgy t the next level in the develpmental cycle. Needs f fllw-n phases cnsidered early in the prgram and included in all f the technical management prcesses. Other Current Practices Questins 24. D yu use the 8 Systems Engineering key questins identified in AFRLI , and listed belw, t supprt the management f yur SBIR prject? Yes r N 25. Did yu knw they existed prir t this survey? Yes r N If n skip t questin On a scale frm 1 t 7 verall d yu find the AFRL 8 Systems Engineering key questins indentified in AFRLI t be value added fr managing yur SBIR prjects? nt useful a little useful smewhat useful mderately useful useful very useful extremely useful Please explain in shrt detail: 78

92 27. Hw useful d yu find each f the 8 SE key questins fr management f yur SBIR prjects? Please scre each questin belw using the scale prvided. (Circle ne) nt useful a little useful smewhat useful mderately useful useful very useful extremely useful Scre 9. Wh is yur custmer? 10. What are the custmer s requirements? 11. Hw will yu demnstrate yu have met the requirements? 12. What are the technlgy ptins? 13. Which is the best apprach? 14. What are the risks t develping the selected technlgy? 15. Hw will yu structure yur prgram t meet requirements and mitigate risk? 16. What is yur business-based transitin plan that meets custmer apprval? 28. Which f the AFRL 8 SE key questins are mst useful? Please explain in shrt detail: 29. Which f the AFRL 8 SE key questins are least useful? Please explain in shrt detail: 30. Is there a Technical Management Prcess Area that is nt addressed in the 8 key questins that yu feel shuld be incrprated? Yes r N If s, please explain in shrt detail: 31. On a scale frm 1 t 7 in yur verall pinin hw much Systems Engineering rigr shuld be applied t SBIR prjects? nne very little mderate detail in gd detail well dcumented very detailed extreme detail Please explain in shrt detail: 32. Is there anything else yu wuld like t add fr his survey? 79

93 Appendix 2: SBIR AF SEAM and AFRL Plicy SE Tasks The belw table captures the SE tasks frm AF SEAM. It illustrates that many tasks frm AF SEAM apply t SBIR hwever many d nt. Apprximately nly 50% f AF SEAM tasks were fund t be either in mst cases r smetimes by the authr. The authr was able t translate the interview results and cmments using his expertise as a SE instructr t identify the AF SEAM tasks fr the SBIR cmmunity In the table belw, Applicable tasks represent that 50% r greater f participants identified it as a valid SBIR task. May be tasks represent that 25% t 50 % f participants identified it as a valid SBIR task. Typically nt tasks represent less 25% f participants identified it as a valid SBIR task. 80

94 Results identified a limited number f SE tasks perfrmed in the SBIR cmmunity fr Cnfiguratin Management. Reviewing SEAM it was evident that mst f the SEAM tasks are fr a mature acquisitin prgram and nt the SBIR envirnment. Table 25: AF SEAM SE Tasks fr SBIR Prjects AF SEAM Cnfiguratin Management (CM) SG ID AF SEAM Specific Gal (SG) Title CMG1 The apprach fr technical baseline management is defined and dcumented. Applicable CMG1P1 Identify accuntability fr the dispsitin f, access t, release f and cntrl Typically nt f the technical baselines. CMG1P2 Establish and maintain plans fr managing the cnfiguratin f the prduct. Typically nt CMG2 Establish and maintain technical baselines while managing change Applicable CMG2P1 Identify the cnfiguratin items and related wrk prducts that will be placed Typically nt under cnfiguratin management. CMG2P2 Establish and maintain cnfiguratin and change management systems. Typically nt CMG2P3 Create r release technical baselines. Typically nt CMG2P4 Track and cntrl changes. Applicable CMG3 Integrity f baselines is established and maintained Applicable CMG3P1 Establish and maintain recrds describing cnfiguratin items May be CMG3P2 Perfrm cnfiguratin audits t maintain integrity f the cnfiguratin Typically nt baselines Results identified several SE tasks perfrmed in the SBIR cmmunity that translated well with Air Frce SEAM. Thse tasks included selecting criteria t be used, methds t be used and cnducting analysis f alternatives. Decisin Analysis (DA) SG ID AF SEAM Specific Gal (SG) Title DAG1 Base decisins n an evaluatin f alternatives using established criteria Applicable DAG1P1 Establish and maintain guidelines t determine which issues are subject t a frmal evaluatin prcess May be DAG1P2 Establish and maintain the criteria fr evaluating alternatives, the relative ranking f these criteria, and select the evaluatin methds Applicable DAG1P3 Identify alternative slutins t address issues Applicable DAG1P4 Evaluate alternative slutins using the established criteria and methds Applicable DAG1P5 Select the slutin(s) frm the alternatives and dcument decisins based n the evaluatin Applicable 81

95 Results identified a limited number f SE tasks perfrmed in the SBIR cmmunity fr Design. Design frm AFRL plicy included Architecture, Interface Management and Integratin Tasks. Results frm the interview identified a lw percentage f SE tasks in these areas being cmpleted. Design (D) SG ID AF SEAM Specific Gal (SG) Title DG1 The design is based upn a dcumented architecture, traceable t Typically nt requirements, and ptimized fr the set f requirements and cnstraints DG1P1 Establish and maintain the architectural design baseline Typically nt DG1P2 Establish and maintain interface designs Typically nt DG1P3 Establish and maintain design artifacts that describe the cnditins, functins, perating mdes, and perating states specific t the cmpnents f the architecture Typically nt DG1P4 Develp ptential prduct-cmpnent slutins, alternatives, and selectin Typically nt criteria DG1P5 Analyze and select prduct-cmpnent slutins that best satisfy the Typically nt established criteria DG2 Develp and dcument a detailed design and implementatin strategy Typically nt DG2P1 Establish initial prduct-cmpnent designs and develpment strategies Typically nt DG2P2 Evaluate whether the prduct-cmpnents shuld be develped, purchased, r Typically nt reused based n established criteria DG2P3 Establish detailed designs fr the prduct-cmpnent Typically nt DG2P4 Establish and maintain a technical data package Typically nt DG3 Assemble the design/develpment prttype(s) in accrdance with the detailed design and integratin strategy May be DG3P1 Establish and maintain the prduct integratin apprach May be DG3P2 Establish and maintain prcedures and criteria fr integratin f the prductcmpnents May be DG3P3 Manage internal and external interface definitins, designs, and changes fr prducts and prduct-cmpnents May be DG3P4 Cnduct, prir t assembly prduct-cmpnent verificatin May be DG3P5 Assemble prduct-cmpnents accrding t the prduct integratin sequence Typically nt and established prcedures 82

96 Results identified several Implementatin SE tasks perfrmed in the SBIR cmmunity. Implementatin is captured under Manufacturing in AF SEAM. Hwever thse tasks identified by AF SEAM d nt identify with the SBIR cmmunity as seen belw since little frm the AFRL Tasks culd be translated int SEAM tasks as seen belw. Manufacturing (M) SG ID AF SEAM Specific Gal (SG) Title MG1 Prepare fr manufacturing May be MG1P1 Establish and maintain strategy and plans fr manufacturing Typically nt MG1P2 Perfrm cncurrent design and manufacturing engineering May be MG1P3 Establish and maintain manufacturing technical data Typically nt MG2 Transitin frm develpment t repeatable and ecnmical prductin at Typically nt desired rate MG2P1 Establish and maintain plans fr transitin t prductin May be MG2P2 Qualify/prf manufacturing prcesses, special tls and test equipment Typically nt MG2P3 Ensure readiness fr manufacturing Typically nt MG3 Manufacture the prduct in accrdance with plans and specificatins Typically nt MG3P1 Ensure that prductin at desired rates is cnducted accrding t the plan Typically nt MG3P2 Establish and maintain inventry and supplier management/cntrl Typically nt MG3P3 Cmplete First Article Inspectin (FAI) Typically nt MG4 Prduct and prcess quality is assessed and imprved Typically nt MG4P1 Establish and maintain piece part cntrl and perfrm manufacturing Typically nt screening MG4P2 Establish and maintain a quality management system Typically nt MG4P3 Establish and maintain defect cntrl Typically nt 83

97 Results fr Technical Planning identified several tasks perfrmed in the SBIR cmmunity that translate t the SE tasks captured under Prject Planning in AF SEAM. Mst are r may be in the SBIR cmmunity as illustrated belw. Prject Planning (PP) SG ID AF SEAM Specific Gal (SG) Title PPG1 Establish and maintain estimates f prject planning parameters Applicable PPG1P1 Define the prject life cycle phases, milestnes, and key decisin pints Applicable PPG1P2 Establish a wrk breakdwn structure (WBS) t rganize the effrt May be PPG1P3 Establish and maintain the scpe f the wrk prducts and tasks that describe the prject cst and schedule Applicable PPG1P4 Establish, validate, and maintain estimates fr cst and schedule Applicable PPG2 Establish and maintain integrated plans May be PPG2P1 Assign respnsibility fr acquisitin and sustainment management, supprt, Typically nt and prduct enhancement PPG2P2 Establish and maintain engineering plans t accmplish prject Applicable PPG2P3 Plan fr the management f prject data Applicable PPG2P4 Plan fr necessary resurces, including persnnel knwledge and skills, t perfrm the prject tasks Applicable PPG2P5 Plan the invlvement f identified stakehlders Applicable PPG2P6 Establish and maintain the technlgy develpment strategy Applicable PPG2P7 Plan fr prduct Reliability, Availability, and Maintainability (RAM) May be PPG2P8 Establish and maintain an Integrated Master Plan and Integrated Master Schedule (IMP/IMS) May be PPG3 Establish and maintain cmmitment t the technical plan Applicable PPG3P1 Review all plans t understand cmmitments and ensure the technical plans and verall plans are integrated and cnsistent May be PPG3P2 Recncile the technical plans t reflect available and estimated resurces Applicable PPG3P3 Obtain cmmitment frm relevant stakehlders respnsible fr perfrming and supprting executin Applicable 84

98 Results fr Requirements Definitin, Analysis and Management identified tasks perfrmed in the SBIR cmmunity that translate t the SE tasks captured under Requirements in AF SEAM. Mst tasks were but sme did nt apply fr SBIR prjects as illustrated belw. Requirements (R) SG ID AF SEAM Specific Gal (SG) Title Stakehlder needs, expectatins, cnstraints, and interface requirements are RG1 cllected and translated int a definitin f needed prduct capabilities/characteristics fr all phases f the life cycle Applicable RG1P1 Elicit stakehlder needs, expectatins, cnstraints, and interfaces Applicable RG1P2 Establish and maintain cncepts f peratins and supprt that define the peratinal capability required Applicable RG1P3 Transfrm stakehlder needs, expectatins, cnstraints, and interfaces int a priritized requirements baseline Applicable RG1P4 Establish and maintain a requirements/decisin data archive t dcument requirements and related technical decisins Applicable RG2 Requirements are refined, elabrated, and allcated t supprt design r service(s) May be Establish and maintain design missin reference prfiles that define the RG2P1 prduct characteristics required in engineering terms and dcument the interactin f the prduct with the envirnment, ther systems and peratinal Typically nt users RG2P2 Allcate the requirements t each prduct-cmpnent Applicable RG3 Iteratively analyze and validate peratinal and derived requirements thrughut the prduct life cycle Applicable RG3P1 Analyze requirements t ensure that they are necessary and sufficient Applicable RG3P2 Analyze requirements t balance stakehlder needs and cnstraints Applicable RG3P3 Validate requirements t ensure the evlving prduct will perfrm as intended in the peratinal envirnment Applicable RG4 Requirements are managed and cntrlled, and incnsistencies with technical Typically nt plans and wrk prducts are identified RG4P1 Use a disciplined prcess fr accepting, vetting, apprving, and prviding Typically nt requirements and changes t the suppliers RG4P2 Establish and maintain cmmitment t the requirements Applicable RG4P3 Establish and maintain bidirectinal traceability between requirements and wrk prducts Applicable RG4P4 Identify and reslve incnsistencies between requirements, prject plans, and wrk prducts Applicable 85

99 Management. Results identified all AF SEAM SE tasks apply t the SBIR cmmunity fr Risk Risk Management (RM) SG ID AF SEAM Specific Gal (SG) Title RMG1 Prepare fr Risk Management Applicable RMG1P1 Determine risk surces and categries Applicable RMG1P2 Define the parameters used t analyze and categrize risks, and the parameters used t cntrl the risk management effrt Applicable RMG1P3 Establish and maintain the strategy and plans t be used fr risk management Applicable RMG2 Identify and analyze risks t determine their relative imprtance Applicable RMG2P1 Identify and dcument the technical risks Applicable RMG2P2 Evaluate and categrize each identified risk using the defined risk categries and parameters, and determine its relative pririty Applicable RMG3 Perfrm risk handling t manage adverse impacts n the prject Applicable RMG3P1 Establish and maintain plans fr mitigating each f the imprtant risks t the prject Applicable RMG3P2 Mnitr and assess risk handling activities Applicable 86

100 Results identified several SE tasks fr Transitin in the SBIR Cmmunity. Hwever nne f these tasks translate t AF SEAM SE tasks fr Transitin, Fielding, & Sustainment. All f these tasks becme relevant in later stages f technlgy develpment and nt during SBIR Phase I & II. SG ID SG1 TFSG1P1 TFSG1P2 TFSG1P3 SG2 TFSG2P1 TFSG2P2 TFSG2P3 TFSG2P4 TFSG2P5 SG3 TFSG3P1 TFSG3P2 SG4 TFSG4P1 TFSG4P2 TFSG4P3 TFSG4P4 TFSG4P5 Transitin, Fielding, & Sustainment (S) AF SEAM Specific Gal (SG) Title Prepare t supprt peratins, maintenance, repair, and dispsal f the prduct Establish and maintain plans fr lgistics supprt f the prduct Establish and maintain the strategy and plan(s) fr transitining acquired prducts int peratinal use and supprt Establish and maintain plan(s) fr the dispsal f the prduct Ensure the resurces, capacity and capability t supprt the peratins, maintenance, repair, and dispsal f the prduct are ready prir t need Establish and maintain budgets fr sustainment activities Establish and maintain prcesses and prcedures fr repair, verhaul, r mdificatin Ensure readiness fr fielding and transitin t peratins and supprt Ensure prduct supprt is maintained during transitin Establish and maintain the required facilities, manpwer, tling and test equipment fr repair, verhaul, r mdificatin Repair, verhaul, r mdify the prduct Repair, verhaul r mdify the prduct in accrdance with established prcedures and prcesses Establish and maintain inventry and supplier management/cntrl t execute the repair, verhaul r mdificatin Maintain Operatinal Safety, Suitability, and Effectiveness (OSS&E) Establish and maintain OSS&E baseline(s) Identify and mnitr safety critical items Identify and mitigate hazards Identify and mnitr peratins and maintenance data Execute (as required) the plan fr decmmissining and dispsal f the prduct Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt Typically nt 87

101 Results fr Technical Assessment identified tasks perfrmed in the SBIR cmmunity that translate t the SE tasks captured under Technical Management & Cntrl in AF SEAM. Mst tasks were except executing supplier agreements since this des nt take place during SBIR Phase I & II but wuld be relevant after the prject transitins. Technical Management & Cntrl (TMC) SG ID AF SEAM Specific Gal (SG) Title TMCG1 Prepare fr Integrated Management Applicable TMCG1P1 Establish and maintain the prject envirnment Applicable TMCG1P2 Establish and maintain supplier agreements Typically nt TMCG1P3 Establish and maintain integrated prduct teams (IPT) May be TMCG1P4 Establish and maintain measurement apprach Applicable TMCG2 Perfrm Integrated Management Applicable TMCG2P1 Mnitr and cntrl the prject in accrdance with prject cmmitments Applicable TMCG2P2 Mnitr & cntrl crdinatin and cllabratin Applicable TMCG2P3 Execute Supplier Agreements Typically nt TMCG2P4 Obtain and analyze specified measurement data Applicable TMCG2P5 Mnitr the develpment and delivery f prject data Applicable TMCG3 Mnitr & Cntrl Technical Prgress Applicable TMCG3P1 Technical reviews and audits are cnducted when all the key entry criteria are met and clsed when the exit criteria are met Applicable TMCG3P2 Assess results f technical reviews t supprt key milestne decisins, higher level reprting, and prject re-planning as required Applicable TMCG3P3 Manage prject wrk prducts and data Applicable TMCG4 Mnitr & Cntrl Crrective Actins Applicable TMCG4P1 Cllect and analyze the prject issues t determine and track crrective actins Applicable TMCG4P2 Establish and maintain a deficiency reprting system Applicable TMCG4P3 Manage crrective actins t clsure Applicable 88

102 Results fr Verificatin and Validatin identified tasks perfrmed in the SBIR cmmunity that translate t the SE tasks captured under Verificatin and Validatin in AF SEAM. Mst tasks were but sme did nt apply fr SBIR prjects since a limited amunt f testing in perfrmed during SBIR Phase I & II. Additinal testing will take place when the prject transitins and thse tasks will be then. Verificatin and Validatin (V) SG ID AF SEAM Specific Gal (SG) Title VG1 Prepare fr verificatin Applicable VG1P1 Establish and maintain the verall verificatin strategy and plan, including integrated testing apprach Applicable VG1P2 Identify the wrk prducts t be verified and the verificatin methds that will be used Applicable VG1P3 Establish and maintain the envirnment and resurces needed t supprt verificatin Applicable VG1P4 Establish verificatin prcedures and criteria Applicable VG2 Peer reviews are perfrmed May be VG2P1 Prepare fr peer reviews f selected wrk prducts May be Applicable VG2P2 Cnduct peer reviews n selected wrk prducts and identify issues resulting May be frm the peer review Applicable VG3 Wrk prducts are verified Applicable VG3P1 Perfrm verificatin n the selected wrk prducts Applicable VG3P2 Analyze and dcument the results f all verificatin activities Applicable VG3P3 Initiate and dcument crrective actins Applicable VG4 Prepare fr validatin Applicable VG4P1 Develp a prduct validatin strategy and identify wrk prducts fr validatin Applicable VG4P2 Establish and maintain validatin criteria, methds and prcedures Applicable VG4P3 Establish and maintain the envirnment and resurces needed t supprt validatin Applicable VG4P4 Ensure apprpriate certificatins & accreditatins have been cmpleted Typically nt VG4P5 Establish and maintain a dcumented plan fr validatin Applicable VG5 Validate prduct t ensure that it will be safe, suitable and effective in the Typically nt intended perating envirnment VG5P1 Perfrm validatin n the selected prducts and prduct-cmpnents May be VG5P2 Analyze and dcument the results f the validatin activities Applicable 89

103 Appendix 3: SBIR SETT Tl Output Research f past wrk n this tpic identified a prir AFIT thesis titled A Tailred Systems Engineering Framewrk fr Science and Technlgy Prjects frm March This thesis built a tl fr identifying what Systems Engineering principals shuld by applied t a particular prgram in AFRL based n inputting the fllwing parameters identified in the example belw: Parameters inputted int tl: Inputted parameters abve wuld fit sme SBIR prgrams. Belw is the utput fr the tl fr thse parameters: 90

104 lse Activity ~ Surce ~ Tls ~ SE Rigr ~ TP l (Requireme nts Devel pme nt) 25% - 100% Establish Cmmunicatins with Stakehlders DAG Sec %. 100% Identify Prj ect Cnstraints DAG Sec %. 100% Determine Required capabilities DAG Sec %. 80% O~ l~r miu ~ O~ sir~t.l P ~r rurrmu 1 t.:~ DAG S" c % 100% TP 2 (Lgical Analysis) 0% - 100% Analysis Preparatin DAG Sec % 100% DDAF OV-5, Perfrm Functinal Analysis DAG Sec SV-5 60% 100% DDAF OV 6(a,b,c), SV Perfrm Behaviral Analysis DAG Sec {a,b,c) 80% 100% Perfrm Envirnmental Analysis DAG Sec %. 100% Design Factrs Analysis INCOSE Pg 4.6 0%. 100% Develp Functinal Architecture DAG Sec %. 100% TP 3 (Desig n S luti n) 10% - 100% Def ine Design Prblem suede Pg 31, 39 20% 100% Generate Alternative Design Slutins Buede Pg 31, 39 10%. 100% Eva luate Design Alternatives DAG Sec %. 80% TP-4 (Impleme ntati n) 5% - 100% Generate Imp lementatin Strategy INCOSE Pg %. 100% Fabricate Hardware DAG Sec %. 80% Cde Sftware DAG Sec %. 100% Cnduct UnitTesting INCOSE Pg %. 100% Cnduct Training INCOSE Pg % - 5% Prepare fr Integratin DAG Sec %. 80% TP 5 (Integrati n) 10% - 100% Determine Integratin Prcess Buede Pg %. 80% Cnduct Assembly / Integratin t System INCOSE Pg %. 100% Re levant Envirnment DAG Sec % 10% TP 6 (Ve rificati n) 30% - 100% Plan Verificatin Buede Pg %. 100% Execute Verificatin INCOSE Pg %. 100% TP 7 (Validati n) 30% - 100% Plan Va lidatin Buede Pg %. 100% Buede Pg 51; Execute Va lidatin INCOSE Pg %. 100% 91

105 lse Activity ~ Surce ~ Tls ~ SE Rigr G TP-8 (Transit i n ) 0% - 100% Identify Transitin Opprtunitie.s INCOSE Pg % - 100% Qualify Prductin It em Suede Pg 314 0% - 0% Execute Transitin INCOSE % - 100% TMP-1 (De cis i n An a ly sis ) 100% - 100% Identify Strategy fr Making Decisin INCOSE Pg % - 100% Execut e Decisin Making Strategy DAG % - 100% TMP-2 (Technica l Plannin g) 0% - 100% Plan Systems Engineering DAG Se< % - 100% Integrated Implement Technical Plan INCOSE Pg Master Plan 30% - 100% Evaluate Plan t Address Needs INCOSE Pg % - 0% TMP-3 (Technica l Assessmen t) 70% - 100% Prepare fr Technical Assessment DAG Se< % - 100% Perfrm Technical Asse.ssment INCOSE Pg % - 100% TMP-4 (Require m e n t s Manageme n t) 5% - 100% Determine Rles/Re.spnsibilit ies During Reqs Generatin Prce.ss Suede Pg % - 100% Define Syst em capabilit ies and Perfrmance Objectives INCOSE Pg % - 100% Validate Requirements Develpment Prce.ss Suede Pg 41 25% - 25% Ensure Requirements Feasibility and Validity Suede Pg 40 5% - 95% INCOSE Pg 4.3, Dcument Requirements % - 100% DAG Se< ; Suede Pg 158; Ensure Traceability f Requirements INCOSE Pg % - 100% Establish Prcess fr Requirements Changes Suede Pg % - 80% 92

106 lse Activity ~ Surce ~ Tls ~ SE Rigr ~ TM P-5 (Risk M anagement) 10% - 100% Risk Management Risk Planning DAG Se< Framewrk 0% DAG Se< ; Suede Pg 314; INCOSE Pg 5.1- Dcumentatin Reviews; Infrmatin Gathering (Brainstrming, De lphi Te, hnique, Interviews, SWOT (Stre ngth Weakness-Opprtunity Risk Identificatin 5.11 Threat) Ana lysis) 30% - 100% 1\RENJ\, CORE, MJ\TLI\8 DAG Se< ; State Flw Mde ler, Crysta l Ba ll (Exce l a dd Risk Analysis (Qualitative & Quantitative) Suede Pg 382 in) 100% - 100% DAG Se< ; Risk Handling INCOSE Pg % - 100% Risk Mnitring DAG Se< % - 100% DAG Se< ; Risk Dcumentatin INCOSE Pg % - 100% TM P-6 (Cnfiguratin M a n a~emen t) 30% - 100% DAG Se< ; INCOSE Pg 4.17 Develp Cnfiguratin Baseline.s INCOSE Pg % - 100% Establish Cnfiguratin Change Cntrl Plan (Establish cnfiguratin cntrl cycle that incrprates evaluatin, apprvat validatin, and verificatin f chcnge DAG Se< ; request.s) INCOSE Pg % - 30% Develp and Maintain Cnfiguratin Cntrl INCOSE Pg 4.6 & Dcumentatin % - 30% DAG Se< ; INCOSE Pg 4.12 & Maintain Cnfiguratin Baselines % - 100% 93

107 lse Activity ~ Surce ~ Tls ~ SE Rigr ~ TMP-7 (Technical Data Management) 10% - 100% DAG Se< ; Cre Architecture Data Develp Data Management Plan INCOSE Pg 5.15 Mdel 30% - 100% Determine I Define System Relevant Infrmatin INCOSE Pg % - 100% Identify System Data t Purchase DAG Se< % - 100% Determine Data Prtectin Requirements DAG Se< % - 100% Addre.ss l ng4 term Data Strage Requirements DAG Se< % - 50% Recrd Prgram Data INCOSE Pg % - 100% Make Prject Data Available INCOSE Pg % - 100% TMP-8 (Interface Management) 10% - 100% Define Interface Requirements and Cntrl Suede Pg 294; Methds INCOSE Pg % - 100% DAG Se< DAG Se< ; Suede Pg 50; Develp System Interface Cntrl Methds INCOSE Pg % - 100% DAG Se< D DAF SV-1, lnte rfa<e Generate Interface Cntrl Dcumentatin DAG Se< Cntrl Dcument 60% - 100% DAG Se< ; Utilize Interface Cntrls Suede Pg 39 10% - 100% Fundamental Principles (Applicable t All PROCESSES) 100% - 100% Utilize Enterprise capabilities INCOSE Pg % - 100% 94