Curda Cafe
EME 6812: IT Seminar

  
The Menu

Book Club Discussions
Weekly Specials

Coffee Chats
Monthly Specials

Email Manager
This Week's Specials

Email Cashier
Cafe Regulars

Email Waitress
Checkout Counter

Email Regulars

Previous Specials

  

Home

Week 4:  June 1

This Week's Menu

 

 Foundations and Research in Instructional Technology

By now, everyone should have posted an original response on each of the three menus from last week. Therefore, this week reserve some time to read your peers' posts and respond to at least 3 on each of the discussions. I will be reading and responding as well this week.

The theme for this week is foundations and research in instructional technology.  Your menus this week (I hope) are going to (1) provide confidence for selecting appropriate research articles for your research project, (2) develop your understanding of many of the different research methods you might come across when reading your articles for your research project, (3) clarify your understanding of what empirical research really is, (4) provide you with guidelines for analyzing and synthesizing empirical research on a topic and (4) actually provide you with a good example of a "review of research" similar to what you are doing in your project but more in depth than what you will accomplish in this short class.  I know that many of these concepts are difficult ones to grasp and I hope that as you read more about things and have more questions that you will feel free to email or call me if you need even more assistance. 

In addition, check out the link for the submission form you will be using for the article analysis portion of research project on the June/July Specials page. I would like to take some time this week to expand on the idea of analyzing research for your research project and what my expectations are for this project.  As I have said somewhere before, critically reading, analyzing, and synthesizing research is a large part of becoming a professional in many fields of study, including IT. Every professor I am sure has an opinion/way in which they feel these skills should be taught - some believe in trial by fire, trial and error, etc.  I generally believe in providing as much guidance as possible.  I do not expect that all of you have had your research and statistics classes yet so I know that reading and analyzing empirical research can be a daunting task.  By not requiring you to write a complete research paper, APA format and all, and to only review 4 articles on your first try I hope that I can eliminate some of the overload so that you can concentrate on the process this involves.  If you are in the online program and are following the suggested course sequence, you will be taking EDF 6218 (the psych course) next semester.  I also teach this course and DO expect you to write a full blown research paper, however, knowing that you have had the introduction that this course provides I rest easier:-) For those of you that have taken the courses out of the sequence then this assignment should be a breeze after having completed the research paper in that course, but I do hope that you will find some of this information helpful. 

The readings for this week are Chapters 4, 5, and 28 (as a review).  In addition, I have provided lots of additional information and links with regards to research paradigms and methodologies for instructional technology research.  The lack of more in-depth information on this topic within your text is the only weakness I have found so far.  Because the information with regards to this is probably new and you do not have the benefit of a neatly written chapter in your text, you may find some of the information difficult to follow.  I have provided numerous links for you to review as needed to develop your own understanding.  You will learn more about all of the topics discussed this week in future courses in your program, including EDF 6218 (ed psych), EDF 5404 (stats), and EDF 6481 (research) so do not fret if you do not grasp it all at this time.

Manager's Special

 

 Main Dish: Psychology & Epistemology (Chapters 4 & 5)

As you begin to read, analyze, and synthesize research in the field of IT or whatever field you are in, you should begin to note the theoretical frameworks from which the researchers approach their study.  The theoretical framework they use will often determine the types of questions they ask, the data they collect, and the results they find. The alignment of research with a theoretical framework is a strength because it provides a scaffold to support the researcher throughout the planning, implementation, and interpretation of his/her particular study.  Clear communication within published research of the theoretical framework on which the researcher drew to plan and conduct the study assist the consumer of that research to better understand the decisions made by the researcher and the interpretation of the results.  As an example, if I wanted to study a group of kindergarten students and their development of mathematical concepts over a semester, I could probably choose to use any of the theoretical frameworks discussed in Chapter 4.  I could choose to look at the rewards, punishments, practice and feedback students are given by their teacher during math lessons.  Or, I could choose to investigate the prior knowledge of each student with regards to particular concepts and identify if certain prior knowledge predicted success in acquisition of particular concepts. Or, I could analyze the classroom environment and the opportunities and materials students are given that allow them to impose organization and meaning to construct mathematical concepts of their own.  There are many other ways to investigate the development of mathematical concepts in kindergarten children.  My point is that the theory or theories you choose to draw from when planning and designing your research will likely determine the direction of your research study. 

As you begin to read research in our field, I would like for you to begin to identify the theoretical framework the researcher draws on for his/her study. Chapter 4 only addresses a few of the most prominent theories at this time.  When you take EDF 6211 (the psych course) you will be exposed to many more (probably more than you ever wanted to know) to choose from.  My concern now is not that you have a complete understanding of every theory (that is left for other courses) but to understand the idea that theories frame how we conduct research -  what questions we ask and how we choose to answer them. Chapter 5 expands on this idea and includes the epistemic beliefs we hold work in concert with the psychological theories we adhere to and the research we conduct to determine the design practices we choose to employ.  In very simplistic terms, the idea is that our beliefs about the nature of knowledge will influence the theories we choose to use when designing instruction.  Even more simple is the idea that most teachers teach using methods that are consistent with the way in which they believe their students will grasp the content most readily.  If a teacher believes reward, punishment, practice and feedback will do the trick then you will likely see instructional activities that employ these techniques. The epistemological beliefs a teacher holds about how students learn will limit the types of instructional methods she will choose. The authors of Chapter 5 do an excellent job of not promoting a single epistemological belief or theoretical framework as "the one" to have when designing instruction or conducting research.  Instead, they argue for "grounded design" practices that require alignment of beliefs, theory, research, and design of instruction. One of my biggest pet peeves is when teachers (or trainers) state that they are "constructivists" but then go on to describe a lesson or training they just developed that does not follow the tenets of the constructivist philosophy. I believe that the current educational climate has forced many to believe that saying you are constructivist is the "politically correct" thing to say.  To say you are a behaviorist might bring you some funny looks of disapproval.  However, too often, I find that one's stated epistemological or theoretical beliefs about the nature of knowledge and the process of learning do not align with what he/she is actually DOING in a classroom. This misalignment only confuses people more.  I am all for teachers, trainers, instructional designers, and others in our field having the freedom to adhere to whatever beliefs they choose given the context they find their self in, but such stated beliefs need to match their actions. I hope reading Chapter 5 brings this point home to you.

Daily Vegetables

If you are especially curious about one or more of the theories discussed in Chapter 4 or in epistemology, I have provided some links here for you to investigate.  Please don't feel like you have to eat all your vegetables (some of you may liken some of these to having to eat beets or spinach or whatever your least favorite vegetable is).  Choose from the list and explore some of your interests.

Epistemology

Behaviorism

Information Processing

Situated Cognition

Constructivism

Enjoy the Meal!

Think about how you would explain what you believe about how people learn to someone else.  Try to write this down in a paragraph or less. Given your statement, what epistemological belief do you most closely adhere to? What theoretical framework(s) most closely align with that belief?  How do you think these beliefs influence how you would design instruction or how you would evaluate an instructional product? How do you think you can insure that you use "grounded design" practices?

In this threaded discussion, share your statement about learning and the answers to the above questions.
Chef's Special Main Dish: Research Paradigms for Instructional Technology

There are many different models, or paradigms, for research in instructional design and technology. Some differentiate between mature science and developing science. In a mature science, one paradigm usually dominates the research studies. A developing science, such as instructional design and technology, draws from research theory of other fields to formulate its own foundation for research and theory development. Due to the developing nature of instructional design and technology, IT researchers often support the idea of embracing a wide variety of research paradigms and I most certainly agree.

Numerous paradigms are available to the IT researcher.  Below is a list of some of these paradigms along with a brief summary of each:

Experiment – It is the most familiar, most used, and most traditional paradigm. Can be used to identify functional laws and cause-effect relationships. Involves experimental control of extraneous variables. Offer the most effective means of establishing factors that influence a phenomenon of interest. Help identify the effects of a single or interacting instructional variable on learning.

Quasi-Experiment – This paradigm is somewhat experimental, but sometimes limiting factors of control prevent strictly experimental conditions. Such factors as those that occur in classroom settings make experimental control impossible or unfavorable. Cook and Campbell (1979) report that despite lack of control, valid causal inferences can result if some effort is given to using statistical and design strategies in field settings.

Meta-Analysis – Meta-analysis is non-experimental and involves combining previously reported research findings. It is used sometimes to determine what the next step of a research effort should be. This type of paradigm can be used to help professionals reach conclusions about the effectiveness of an instructional technology on learning and the impact of the effect.

Case Study/Ethnography – Case study and ethnography paradigms assume that context, the set of circumstances or facts that surround a particular event or setting, in part determines and defines any phenomenon in question. These paradigms are useful in answering “how” and “why” questions pertaining to instructional design and technology. Ethnography is a type of case study that draws from the fields of anthropology and sociology. The “culture” of the teaching-learning environment is the focus of this paradigm. Processes employed in this type model are observation, interviewing, and content analysis of human artifacts. Factors that limit the use of this type paradigm include the extent of time involved and the great volume of pages to report.

Systems-Based Evaluation – This paradigm is used to determine what makes technologies work in some settings and not in others. It explores social, political, and economic influence on the effectiveness of instructional systems.

Cost-Effectiveness – Cost-effectiveness models are implemented to establish if one method, procedure, or program is more or less expensive than another. It is important to define and consider other criteria besides cost when judging the effectiveness of a program, technique, or procedure. Researchers in other countries more often use this model.

Technique and Model Development – New or revised models of instructional design and new instructional techniques for analyzing and presenting content are the focal points of this paradigm. This is warranted by the increased diversity of learning environments and more problematic content.

More than one paradigm can be used. Combining paradigms would increase the possibility of solutions. Considering the developing nature of the science of instructional design and technology, we should try not to limit our view of research paradigms. If we limit our view of research paradigms more questions will go unanswered. Instead we should utilize a variety of research paradigms to broaden our perspective.

As you search for research articles to use in your research project many of you will find a wide variety of articles using varied methodologies.  Some of the articles will use more than one methodology to answer their questions.  As noted above, this is a good thing.  Always using the same research methodology can sometimes lead to stagnate research that does not answer questions of interest or value in a particular area.  It used to be that only strict quantitative research was accepted in fields that wanted a good reputation as a science.  Through years of hard work, however, qualitative research has been given a good status in many fields, including ours.

Qualitative research is typically an in-depth analysis of relatively few subjects whereas quantitative research applies statistics to a large number of subjects. When properly applied, quantitative research is more statistically powerful than qualitative studies but not necessarily more relevant. Quantitative studies do have their drawbacks however. Many studies are poorly designed, statistics can be biased, and significant statistical differences may not be discovered due to small sample sizes. Also, statistics can be misinterpreted and the subsequent studies infected. Just because it is published does not necessarily make it great research (you will come to realize this more and more once you being analyzing research for your project).

 Over the past 40 years, the advances in social sciences and educational technology have merged with the application and interpretation of qualitative studies, particularly over the last decade. Past educational technology research was dominated by testing and assessment, comparison and contrast. Qualitative research has slowly moved into the realm of acceptable research methodologies. The most fundamental shift in research in the last 30 years is a move from quantitative science of evaluation and experimental research methods based on standardized test scores. Instructional technology is complex, hence the research surrounding it is also complex. Many IT researchers discuss the need for qualitative research in instructional technology to be multi-faceted and have multiple perspectives, and reflect social change. Complete qualitative research may contain quantitative data whereas qualitative research is frequently rejected from a quantitative approach. Changes have been phenomenal in the acceptance and development of qualitative research in instructional technology. For example, when the Yahoo search engine searched for "Qualitative Research in Educational Technology", over 35,000 sites were identified. Another sign of professional acceptance of qualitative research is highlighted by the Association for Educational Communication and Technology (AECT) annual award recognizing outstanding qualitative research studies and publications. Higher education is also accepting qualitative research with some institutions priding themselves on that development.

An area that has seen tremendous growth in qualitative research is the development of software available to assist in data organization, retrieval, and analysis. Computer Assisted Qualitative Data Analysis Software (CAQDAS) is a UK networking project to provide support and software information for the use of computer programs designed for qualitative data analysis. This site also provides free down-loads and software review of programs such as QSR NUD*IST, NvIVO, ANSWR, CTANKS, and HyperResearch 2.0. Ethnograph 5.0 is another program available at www.qualisresearch.com. The development of this myriad software for qualitative research is indicative of the level of professional acceptance for this research methodology.

Publication using qualitative research doesn't seem to be as difficult as a decade ago but good qualitative research is still in its infancy for publication. Qualitative research still has many drawbacks that must be overcome. One of the biggest issues facing qualitative research is that it is very time consuming and expensive. Interviews and observations can also be plagued by interviewer bias. Another problem is qualitative research does not have a single accepted methodology outline. At this point, there is not qualitative standardization to aid researchers in ensuring publication and acceptance of the research methodologies. The final issue is the "selling" of the qualitative research outside of academia. Politicians are accustomed to making decisions on hard facts and numbers. Qualitative research doesn't necessarily provide that level of knowledge and frequently brings about more questions than answers.

So where is the future of qualitative research in educational technology? U.S. Secretary of Education, Richard W. Riley said at the 1999 National conference on Educational Technology, "We are far enough along in the technology revolution and its application to learning that it is time for a systematic review and analysis of what works best." Qualitative research is now an accepted methodology but requires a high level of detail and organization to be pertinent. Standardizing research methods will aid researchers in providing quality material. Software advances will also assist in the standardization of analysis ensuring the reliability and validity for global acceptance. Finally, qualitative research methods will involve universities, industry, and the public schools as partners in the development of state of the art evaluation techniques and long-term research projects.

For a quick example of a qualitative study, look at Worschauer's paper on Technology and School Reform. It addresses the critical issues in evaluating the effectiveness of educational technology utilizing a qualitative approach to research.

As you think about these paradigms in the context of your research project, I hope that you can reflect on what you are reading and begin to identify the specific methodology being used. Have fun!

Daily Vegetables (A local farmer's crop came in and boy did he have some variety - enjoy!)

Ethnography/Case Study Links 

Experiment/Quasi-Experiment Links

Research Methods Links

Qualitative links

Enjoy the Meal!

A variety of paradigms for research are illustrated in this menu. Select one or more methods and give a specific example of a research study you could conduct to answer a question in instructional technology that would use the method(s) you choose. OR, identify an empirical research study available on line and provide us with the url as well as a summary of the research question and the paradigm used.  Reply to at least 3 of your peers' ideas for research (next week). Place responses on this threaded discussion.
Soup and Salad Special
 		 Conducting and Reading Instructional Technology Research

Our soup and salad special this week (I hope) will provide you with early information you may need to clarify your understanding of what empirical research is and the expectations I have for your research project (important since that is coming around quickly:-).

Most any refereed empirical research article published in our field can be recognized by the components that many IT researchers suggest as steps in planning your research.  Put simply, empirical research is a kind of research in which the investigator poses a question to answer and seeks to answer it by gathering and analyzing data. The data need not be numerical, nor need it be directly observed by the investigator.  They must only be products of observation.  Thus, an ethnographer who makes a written claim about some practice of a native tribe is an empirical researcher, though the claim is expressed in words.  Similarly, a physicist who measures properties of a subatomic particle he or she has never seen, and then makes a claim about that particle expressed numerically, is doing empirical research.  The important thing is that both claims can be rendered false by further research, and both claims were supported through collecting and analyzing data using accepted methodologies. The method by which they collected their data may be very different but similarities exist across the board that can help you identify empirical research.  These similarities include (1) a research problem or question, (2) a review of existing research literature on the problem or question, (3) research questions or hypotheses, (4) a research design that guides the collection and analysis of the data, (5) methods related to identifying/selecting subjects, materials and instruments to be used, and other procedures that are followed, and (6) final analysis of the data collected and a report of the results and discussion of how the research question(s) can be answered given the data collected and analyzed. These are the key components you should be searching for when attempting to find empirical research on your topic so you should make sure you have a good grasp on these components.

Once you find an empirical research article the next question you need to ask yourself is  - how well was this research conducted and is it worthy of being included in my review of the research topic I chose?  The requirement that your article is in a refereed journal helps you in this aspect because at least you know that other scholars in the field have deemed it good enough for publication (although you may be surprised at what gets published once you analyze it carefully). However, you will still need to read it carefully to insure it fits well with your topic. Once you decide it fits with your topic, your job is to analyze the article and write a summary report of your impression.  Analysis of an empirical article involves more than just summarizing the contents in your own words (something everyone is usually guilty of when they first begin writing "research" papers).  Instead, it involves making a critical judgment of the quality of the research conducted and the value and application of the findings. As you read the article you will need to think critically about how well the article addresses the six components mentioned above. As you do this there are some guiding questions you can ask yourself as well as jot down notes for later inclusion in your article analysis. These guiding questions are:

Research problem/question:

  • Is the problem an important/relevant one to the field of instructional technology?
  • Will this study contribute to our understanding of the problem?
  • Do they provide a good rationale for why such a study is important?

Literature review:

  • Does the literature review provide definitions of the key constructs they are investigating?
  • Do the authors critically evaluate the studies they review?
  • Does the literature provide support that this study is needed?

Research questions/hypotheses:

  • Are the questions/hypotheses clearly stated?
  • Will the answer(s) to the question(s) be applicable to practice in the field of IT?
  • Will the answer(s) to the question(s) help us understand the problem better?

Research Design and Methods (components 4 and 5 combined):

  • Is the sample (subjects) they used appropriate for answering the questions?
  • Can you generalize findings to a larger population? Will you be able to apply their findings to similar settings?
  • Were the instruments or measures they used clearly identified/explained?
  • Do the authors convey clearly the procedures used for collecting and analyzing the data?

Results and Discussion:

  • Are the analysis techniques they used (whether quantitative or qualitative) clearly explained?
  • Do the findings they report seem plausible given the data they analyzed? Do they make sense?
  • Do the conclusions they make follow logically from the data analysis findings?
  • Do the conclusions they make match the questions they posed at the beginning of the article?
  • Do the authors report limitations to their study?
  • Do the authors provide important implications for practice?
  • Do the authors make suggestions for future research in this topic?

These components and their guiding questions will be the areas on which you will concentrate when completing your article analyses.  You will be expected to summarize each of the components as stated in your article as well as provide a critical judgment of how well they communicated and conducted their research using the guiding questions. I have provided a web-based form for you to begin submitting your article analyses if and when you are ready.  Remember that the first 2 analyses are due June 29th.  I hope this article as well as my summary above assists you in successfully completing this assignment.

If you would like an even better idea of how others in the field summarize, analyze, critique, and synthesize others' research I strongly suggest you pick of an issue of the Review of Educational Research (RER) and find one of the articles in there to read.  This journal focuses completely on comprehensive reviews of research on specific topics within education.  This is much like what you are doing with your research topic but on a much smaller scale.  You are only expected to include 4 articles in your literature review while an article in this journal would be somewhat exhaustive, reviewing every research study completed on the topic.  However, if you take a look at some of the individual subheadings in the review you are likely to find one or more models of how your analyses and synthesis will eventually develop. Most often, the authors provide a nice summary and critique of the major studies in this area and note the major strengths and weaknesses, how the studies are similar or different, and what still needs to be done.  I hope reviewing an article within RER provides you with a model for how researchers analyze, critique and synthesize research in the field.

Daily Vegetables

If you are still unsure about whether or not you are able to identify empirical research I think a good litmus test is whether or not you can answer the guiding questions (positively or negatively).  If there are no subjects then it is likely not empirical.  If there are no research questions/hypotheses, then it's empirical nature is questionable.  Do you see where I am going with this? If you are having an extremely difficult time identifying empirical research then my next question would be, are you only looking online?  If the answer is yes, then the solution is find a library to go to and search there.  As I have stated before, while some may luck out, many will have to actually go beyond what is available on your desktop or laptop to meet the requirements.  Sometimes this will be because of the topic you choose, but mostly it will be because of the still prevalent lack of research published on the web that is both empirical and refereed. However, just because it meets both criteria does not mean it will be a great article (insightful, engaging, applicable, etc.), so you can imagine how bad it can get without meeting these criteria. While it would be wonderful if you found 4 great articles - this is not the point of the project. If there were only great articles there would be no need for close analysis. Listed below are some links that may help but not every link will lead you to complete lists of empirical and refereed journals - you still have to investigate carefully.  However, I thought these might assist you.

Enjoy the Meal!

Article Analysis Practice Activity

For this activity I would like for you to take a stab at analyzing an article in at least 2 of the 5 components listed above. I am providing you an article on-line that is empirical and refereed to read and critique. The article is  The Power of Two: A Study of the Integrated Reading Method of Peer Tutoring.  

In this threaded discussion, choose two of the five components to summarize and analyze on this article.  In your subject line report which component you are working on.  Hopefully, this will allow you to see how others are completing the assignment and will provide me with a way to provide you with some initial feedback to your approach on your research project. Next week I will provide you with my own personal analysis for you to compare.

Call on the Cafe Crew

 Need Help?  Have a question but can't find the answer? Here are your options:
 Please proceed to the checkout counter -
Don't forget to check out next week's specials!