OpenGL stencil not working...

[Vaclav_]

Please read carefully. I am not looking for RTFM replies. I am getting nowhere in OpenGL forum. I do understand this is NOT OpenGL forum. I am NOT looking for fancy shading graphics, I am not looking for how to retrieve object outline either. I am looking for as detailed as possible practical flow in using OpenGL stencil. I have bits and pieces and it is all just "try this ... try that..." I am looking for help, not for pointless criticism of my coding style. If you cannot help me , please dot not reply. glPushAttrib( GL_ALL_ATTRIB_BITS); // must clear all for resizing // TOK resizing works ! glClear( GL_STENCIL_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_COLOR_BUFFER_BIT); glClearStencil(0); // why and what ??? glEnable( GL_STENCIL_TEST); // TOK // glStencilMask is set to 1 as default (??) //glStencilMask(0x00); // no effect //glStencilMask(0xFF); // no effect { // process object block is this correct place ?? OpenGL_Render_RealAxis(); // int parameter = 0) OpenGL_Render_Plus_IMG_Axis(); //int parameter = 0) OpenGL_Render_Minus_IMG_Axis(5); //passed temporary last circle as parameter 5 } // TOK turn off writing to color buffer - do no show object to be stenciled glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Set the stencil buffer to write a 1 in every time // a pixel is written to the screen glStencilFunc( GL_ALWAYS, 1, 0xFFFF); glStencilOp( GL_KEEP, GL_KEEP, GL_REPLACE); { // process block dupe OpenGL_Render_RealAxis(); // int parameter = 0) OpenGL_Render_Plus_IMG_Axis(); //int parameter = 0) OpenGL_Render_Minus_IMG_Axis(5); //passed temporary last circle as parameter = 0) } parameter_stencil = 42; // verify stencil function parameter_stencil = Stencil(parameter_stencil); if (parameter_stencil == 0) { printf( "\nTOK HERE Stencil full circle parameter %i @function %s @line %i", parameter_stencil, __FUNCTION__, __LINE__); glStencilFunc(GL_EQUAL, 1, 1); } // show / keep inside circle else { printf( "\nStencil last circle parameter %i @function %s @line %i", parameter_stencil, __FUNCTION__, __LINE__); glStencilFunc(GL_NOTEQUAL, 1, 1); } // keep all outs

[leon de boer]

Well you failed the first part of the flow you need to enable the stencil buffer before you clear it :-)

glEnable(GL_STENCIL_TEST); // ENABLE the stencil buffer
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); // CLEAR the stencil buffer along with colour and depth

From there the steps are - Enable writing to the stencil buffer. - Render objects, updating the stencil buffer. - Disable writing to the stencil buffer. - Render other objects - display whole result

In vino veritas

[Vaclav_]

I am sorry, but ALL buffers - display modes (?) are enabled in main OpenGl "event loop". I though that glEnable(GL_STENCIL_TEST); // ENABLE the stencil buffer is specifically to enable writing to the stencil buffer. Am I wrong ?

window = glutCreateWindow("OpenGL - Base "); // Create window with the given title
// single for now
glutInitDisplayMode(GLUT\_SINGLE | GLUT\_RGB | GLUT\_DEPTH | GLUT\_STENCIL);

I think I need to review all in old-fashioned "IBM flow chart" style - from top to bottom - from "vertex" to pixel on screen. This OpenGL "pipe architecture" - from left to right - is just not working when it comes to stencilling. What is confusing is mixing of hardwareand software. "stencil works on pixel(s) " Fore example "stencil mask "defaults" to 1" then I see stuff like this glStencilMask(0xFFFF); I may also have modlelview stack messed-up.

[Vaclav_]

Here is my latest code. I'll try to delete my debugging stuff. Hope I 'll not delete something important. The code basically works, it does apply the stencil - to show only data INSIDE the stencil circle. BUT ONLY when I enable viewing of the stencil AND data being written to the color buffer. I have highlighted the code. I was under the impression this is ONLY for viewing the stencil, but apparently not. Obviously I do not have to re-able the color buffer , hence I get BOTH - stencil and the stencilled "fragment". I have no idea how to proceed and where is my error.

  //
		glClear(
		GL\_STENCIL\_BUFFER\_BIT | GL\_DEPTH\_BUFFER\_BIT | GL\_COLOR\_BUFFER\_BIT);

		glDisable( GL\_STENCIL\_TEST);
		glEnable(GL\_STENCIL\_TEST);
		// clear all three buffers
		glClearStencil(0x0);
		glClear(
		GL\_STENCIL\_BUFFER\_BIT | GL\_DEPTH\_BUFFER\_BIT | GL\_COLOR\_BUFFER\_BIT);
		glStencilMask(GL\_TRUE); // MUST enable writing to stencil buffer (??)
		**// turn off writing to color buffer
		glColorMask(GL\_FALSE, GL\_FALSE, GL\_FALSE, GL\_FALSE);
		// enable only to verify the stencil
		//glColorMask(GL\_TRUE, GL\_TRUE, GL\_TRUE, GL\_TRUE);**
		// setup stencil function
		// function  - test always are ref and mask bogus , ignored ?
		// referrence
		// mask
		glStencilFunc(GL\_ALWAYS, 1, 1);							

		// setup stencil operation
		//glStencilOp(GL\_REPLACE, GL\_REPLACE, GL\_KEEP);			// ??
		// blank here

		//glStencilOp(GL\_KEEP, GL\_KEEP, GL\_REPLACE);
		// get stencil
		// TODO need parameter here to verify 
		parameter\_stencil = 42;
		parameter\_stencil = Stencil(parameter\_stencil);		// parameter);
		glFlush();
		// blank here
		// switch keep area
		if (parameter\_stencil == 0) {
			printf(
					"\\nStencil full circle parameter %i @function %s @line %i",
					parameter\_stencil, \_\_FUNCTION\_\_, \_\_LINE\_\_);
			glStencilFunc(GL\_EQUAL, 1, 1);
		}	// keep inside circle
		else {
			printf(
					"\\nStencil last circle parameter %i @function %s @line %i",
					parameter\_stencil, \_\_FUNCTION\_\_, \_\_LINE\_\_);
			glStencilFunc(GL\_NOTEQUAL, 1, 1);
		}	// keep all outside of stencil
	


		{
			// apply stencil 
			//int OpenGL\_Render\_RealAxis(int parameter = 0)
			//OpenGL\_Render\_RealAxis(); // int parameter = 0)
			OpenGL\_Render\_Plus\_IMG\_Axis(); //int parameter = 0)
			//OpenGL\_Render\_Minus\_IMG\_Axis(5); //passed temporary last circle as parameter = 0)
		}

from here I am not sure how to display only the rendered frag